atom feed37 messages in org.freebsd.freebsd-portsRe: [ECFT] pkgng 0.1-alpha1: a replac...
FromSent OnAttachments
Baptiste DaroussinMar 25, 2011 3:10 am 
Alexander LeidingerMar 25, 2011 7:06 am 
Baptiste DaroussinMar 25, 2011 7:14 am 
Alexander LeidingerMar 25, 2011 7:37 am 
Julien LaffayeMar 25, 2011 8:02 am 
Pietro CeruttiMar 25, 2011 8:34 am 
Baptiste DaroussinMar 25, 2011 8:41 am 
Andriy GaponMar 25, 2011 9:35 am 
Michel TalonMar 25, 2011 9:44 am 
Michel TalonMar 25, 2011 9:47 am 
Eitan AdlerMar 25, 2011 9:54 am 
Thomas DickeyMar 25, 2011 12:32 pm 
Alexander LeidingerMar 25, 2011 1:14 pm 
Jos BackusMar 25, 2011 1:24 pm 
Garrett CooperMar 25, 2011 1:46 pm 
YuriMar 25, 2011 2:31 pm 
Baptiste DaroussinMar 25, 2011 2:37 pm 
Alexander LeidingerMar 25, 2011 2:52 pm 
Marcin WisnickiMar 25, 2011 4:00 pm 
Baptiste DaroussinMar 26, 2011 3:22 am 
Marin Atanasov NikolovMar 26, 2011 5:38 am.patch
Marcin WisnickiMar 26, 2011 6:18 am 
Michel TalonMar 26, 2011 6:18 am 
Julien LaffayeMar 26, 2011 4:01 pm 
Andriy GaponMar 28, 2011 10:43 am 
Garrett CooperMar 28, 2011 10:58 am 
Julien LaffayeMar 28, 2011 11:22 am 
Benjamin KadukMar 28, 2011 8:30 pm 
Tim KientzleMar 28, 2011 9:15 pm 
Baptiste DaroussinMar 28, 2011 10:50 pm 
Julien LaffayeMar 29, 2011 5:11 am 
Super BisquitMar 29, 2011 7:50 am 
Andriy GaponMar 29, 2011 10:37 am 
Baptiste DaroussinMar 29, 2011 11:27 am 
Baptiste DaroussinMar 29, 2011 1:29 pm 
Andriy GaponMar 31, 2011 7:54 am 
Baptiste DaroussinMar 31, 2011 8:01 am 
Subject:Re: [ECFT] pkgng 0.1-alpha1: a replacement for pkg_install
From:Garrett Cooper (yane@gmail.com)
Date:Mar 25, 2011 1:46:50 pm
List:org.freebsd.freebsd-ports

On Fri, Mar 25, 2011 at 1:14 PM, Alexander Leidinger <Alex@leidinger.net> wrote:

On Fri, 25 Mar 2011 16:35:21 +0100 Pietro Cerutti <ga@freebsd.org> wrote:

On 2011-Mar-25, 15:03, Julien Laffaye wrote:

What about DB corruption/loss? Do you keep the /var/db/pkg/<package>/xxx files even with pkgng and only use the DB as a way to speed up some work (so the DB corruption just requires to run pkg2ng), or are you lost of the DB is lost?

Nothing is done about DB corruption/loss, I am not sure we need to do something. Maybe.

I would say "for sure". Info: In Solaris 10 sqlite is used for the service managenemt framework (SMF). It is possible that the DB is corrupt in some bad situations. In this case you have to rebuild the DB (script provided, been there, had to use it).

If sqlite is properly used with transactions, it is very hard to corrupt the database. But if hardware lies to us and say that the

And as I told above, I even had such a case (more than once), and the hardware was not buggy. What do you want to tell in this case, "life sucks, reinstall everything"?

If you use binary packages, pulling down everything should be trivial, fast, and easy to install. If you're using ports, well then things are going to be slow as expected.

data is on disk whereas it isnt... what can we do?

Sometimes you have to stay with broken hardware.

Sometimes you have to go buy new parts?

Playing with broken hardware is like playing with fire -- sometimes you'll get burned if it goes out of commission during critical operations. I would be more concerned about overall system operation than having a packaging system that can handle all error conditions that should be rightfully handled by various kernel subsystems. If the kernel's doing it's job, then the packaging manager can do its job as well.

Another potential problem is fsync(), but if it is broken on FreeBSD we want to fix it!

BTW, the goal is to only have the database and not the flat files. If you are paranoid about power outage, use something like zfs snapshots...

There are more FS than only ZFS (personally I use ZFS, and I have snapshots, but this is not a good solution for this problem).

A lot of filesystems feature snapshot'ing, including UFS. If you aren't smart enough to back up your data you're toast if the data is gone.

I would be more concerned about the program getting killed, not getting properly cleaned up, etc as this is something that the package manager frontend (or whatever the official name is) should catch and fail gracefully with. Things need to follow an ACID methodology and be recoverable in the event that it can't be ACID, or it's no better than pkg_install/ports currently is if it's caught in the middle of a critical operation today installing or removing software.

If SQLite can't deliver this level of ACID-like capability, then pkg_install needs to be redesigned.

As I told already, if it isn't automatic, nearly nobody will use it. And the package management stuff has to be automatic, no freshman will think about setting up a snapshot script when he starts to use packages/ports.

I'd just provide an export command to print out a <blah> (JSON?) version of the information, and move on. None of the other major packaging systems out there that I know of use flat files for this data, and I would rather not make it automatic because it's an unnecessary performance hit. If the user feels the need for backing up his/her data they will. If not, they're SoL in the event of a crash.

No need to look for strange scenarios, I'm surely going to sudo rm -f the file more sooner than later, so... maybe just save a copy?

A copy or two would be enough, but it has to be done automatically, and once a day is not enough. A copy after each X modifications maybe (for suitable definitions of X and 'modifications').

Please see my comment above. There's no reason why this belongs in a packaging system (you can add it as an external tool, but the point is to avoid architectural mistakes that leaked into the old pkg_install over the period of 10 years or so).

Thanks, -Garrett

PS Sorry for being so hardnosed on this, but I want something that's fast and correct, instead of something bloated, slow, half-baked, harder to test, etc. pkg_install gets executed enough times during a port upgrade that having something more streamlined for most usecases is the only way to go, and there's enough code that doesn't get executed on a regular basis that has no business being in pkg_install.