Feature #7157

Add duplicity (Encrypted bandwidth-efficient backup using the rsync algorithm)

Added by Predrag Zečević over 2 years ago. Updated over 2 years ago.

Status:NewStart date:2016-07-01
Priority:NormalDue date:
Assignee:Predrag Zečević% Done:

0%

Category:OI-Userland
Target version:Hipster
Difficulty:Medium Tags:

Description

Hi,

I have tried to get duplicity (http://duplicity.nongnu.org/index.html) working on /hipster:
  1. Had to install prerequisite package:
    pfexec pkg install -v library/librsync
    
  2. Got latest duplicity downloaded:
    wget https://code.launchpad.net/duplicity/0.7-series/0.7.07.1/+download/duplicity-0.7.07.1.tar.gz
    tar xf duplicity-0.7.07.1.tar.gz
    cd duplicity-0.7.07.1
    pfexec /usr/bin/python2.7 setup.py install --prefix=/opt/SFW
    
  3. Fixed python call (to use python2.7):
    pfexec vim /opt/SFW/bin/duplicity
    
  4. Calling produces error:
    $ duplicity -V
    Traceback (most recent call last):
      File "/opt/SFW/bin/duplicity", line 45, in <module>
        from lockfile import LockFile as FileLock
    ImportError: No module named lockfile
    

    Looked after that error on google and in most cases some "python-lockfile", or "py27-lockfile", etc package was fix.
    Searched around and could not find it in OI (/hipster). Is it complicated to add one (e.g. python lockfile)?
    From the other side, https://pypi.python.org/pypi/lockfile suggests that it is obsoleted and one has to use https://pypi.python.org/pypi/fasteners or http://docs.openstack.org/developer/oslo.concurrency/
    • Fastener is already there:
      $ pfexec /usr/bin/pip-2.7 install fasteners
      Requirement already satisfied (use --upgrade to upgrade): fasteners in /usr/lib/python2.7/site-packages
      
    • Installing oslo.concurenty is as simple as:
      pfexec /usr/bin/pip-2.7 install oslo.concurrency
      
    • Even tried to add lockfile:
      $ pfexec /usr/bin/pip-2.7 install lockfile
      

      but neither does help... Somehow I am lost here.

BTW, tried also with default python - no changes...

Maybe someone else knows how to get this done properly?

Regards.

History

#1 Updated by Predrag Zečević over 2 years ago

UPDATE:

I have created backup BE and installed duplicity (and fixed python call) to system regular path with:

pfexec /usr/bin/pip install lockfile
pfexec /usr/bin/python setup.py install 
pfexec vim /usr/bin/duplicity # fix python call: python2 --> python2.6

And now it does not complain.... I need to to check if it is working properly.
$ /usr/bin/duplicity -V
duplicity 0.7.07.1

Would be nice then, to have lockfile installed before adding duplicity.

I will update ticket once I do some tests.

Regards.

#2 Updated by Aurélien Larcher over 2 years ago

  • Tags deleted (needs-triage)
  • Category set to OI-Userland
  • Assignee set to Predrag Zečević
  • Target version set to Hipster

#3 Updated by Aurélien Larcher over 2 years ago

Any update? Can you prepare the component?

#4 Updated by Predrag Zečević over 2 years ago

Will update soon

#5 Updated by Predrag Zečević over 2 years ago

There is new version available, so tested in zone:

  • Install prerequisites:
    $ pfexec pkg install -v library/librsync network/ftp/lftp crypto/gnupg
    : # ...
    
    $ pfexec /usr/bin/pip install lockfile
    Downloading/unpacking lockfile
      Downloading lockfile-0.12.2.tar.gz
      Running setup.py egg_info for package lockfile
    
        Installed /tmp/pip_build_root/lockfile/.eggs/pbr-1.10.0-py2.7.egg
        [pbr] Processing SOURCES.txt
        warning: LocalManifestMaker: standard file '-c' not found
    
        warning: no previously-included files found matching '.gitignore'
        warning: no previously-included files found matching '.gitreview'
        warning: no previously-included files matching '*.pyc' found anywhere in distribution
    Installing collected packages: lockfile
      Running setup.py install for lockfile
        [pbr] Generating AUTHORS
        [pbr] AUTHORS complete (0.0s)
        [pbr] Reusing existing SOURCES.txt
    Successfully installed lockfile
    Cleaning up...
    
    $ which python
    /usr/bin/python
    
    $ python --version
    Python 2.7.12
    
  • Get source
    $ wget https://code.launchpad.net/duplicity/0.7-series/0.7.10/+download/duplicity-0.7.10.tar.gz
    $ tar xf duplicity-0.7.10.tar.gz
    $ cd duplicity-0.7.10
    
  • Install:
    $ pfexec python setup.py install
    running install
    running build
    running build_py
    creating build/lib.solaris-2.11-i86pc.32bit-2.7/testing
    copying testing/__init__.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing
    copying testing/test_code.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing
    creating build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
    copying testing/functional/test_cleanup.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
    copying testing/functional/test_log.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
    copying testing/functional/test_restart.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
    copying testing/functional/test_verify.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
    copying testing/functional/test_selection.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
    copying testing/functional/test_final.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
    copying testing/functional/__init__.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
    copying testing/functional/test_rdiffdir.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
    copying testing/functional/test_badupload.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
    creating build/lib.solaris-2.11-i86pc.32bit-2.7/testing/overrides
    copying testing/overrides/gettext.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/overrides
    copying testing/overrides/__init__.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/overrides
    creating build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_lazy.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_gpginterface.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_file_naming.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_backend_instance.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_statistics.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_manifest.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_collections.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_dup_temp.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_dup_time.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_gpg.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_backend.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_globmatch.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/__init__.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_tarfile.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_tempdir.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_path.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_patchdir.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_selection.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    copying testing/unit/test_diffdir.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
    running build_ext
    running build_scripts
    running install_lib
    running install_data
    running install_egg_info
    running egg_info
    writing requirements to duplicity.egg-info/requires.txt
    writing duplicity.egg-info/PKG-INFO
    writing top-level names to duplicity.egg-info/top_level.txt
    writing dependency_links to duplicity.egg-info/dependency_links.txt
    reading manifest file 'duplicity.egg-info/SOURCES.txt'
    writing manifest file 'duplicity.egg-info/SOURCES.txt'
    removing '/usr/lib/python2.7/site-packages/duplicity-0.7.10-py2.7.egg-info' (and everything under it)
    Copying duplicity.egg-info to /usr/lib/python2.7/site-packages/duplicity-0.7.10-py2.7.egg-info
    running install_scripts
    changing mode of /usr/bin/rdiffdir to 755
    changing mode of /usr/bin/duplicity to 755
    
  • Fix python call:
    $ pfexec sed -i 's#env python2#env python#' /usr/bin/duplicity
    $ duplicity --version
    duplicity 0.7.10
    
    $ man duplicity
    cat: /usr/share/man/man1/duplicity.1: Permission denied
    ()                                                                          ()
    
                                  September 15, 2016                            ()
    $ ls -al /usr/share/man/man1/duplicity.1
    -rwx------ 1 root root 65611 Aug 20 21:13 /usr/share/man/man1/duplicity.1
    $ pfexec chmod 664 /usr/share/man/man1/duplicity.1
    

    Will do test backup and paste findings here

#6 Updated by Predrag Zečević over 2 years ago

Test with ftp sync (it expects 'gpg' binary, not 'gpg2' as we have):

$ FTP_PASSWORD=my@ftp duplicity --gpg-binary /usr/bin/gpg2 duplicity-0.7.10.tar.gz ftp://ftp@solarix:/incoming
LFTP version is 4.3.8
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase: 
Retype passphrase to confirm: 
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1473926013.01 (Thu Sep 15 09:53:33 2016)
EndTime 1473926013.59 (Thu Sep 15 09:53:33 2016)
ElapsedTime 0.58 (0.58 seconds)
SourceFiles 1
SourceFileSize 1543523 (1.47 MB)
NewFiles 1
NewFileSize 1543523 (1.47 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1
RawDeltaSize 1543523 (1.47 MB)
TotalDestinationSizeChange 1546442 (1.47 MB)
Errors 0
-------------------------------------------------

Check for files:
$ ls -al /export/Ftp/incoming/duplicity-full*
-rw-r--r-- 1 ftp nogroup  36K Sep 15 09:53 /export/Ftp/incoming/duplicity-full-signatures.20160915T075329Z.sigtar.gpg
-rw-r--r-- 1 ftp nogroup  217 Sep 15 09:53 /export/Ftp/incoming/duplicity-full.20160915T075329Z.manifest.gpg
-rw-r--r-- 1 ftp nogroup 1.5M Sep 15 09:53 /export/Ftp/incoming/duplicity-full.20160915T075329Z.vol1.difftar.gpg

#7 Updated by Predrag Zečević over 2 years ago

Test with rsync (only to rsync server - so no user/password needed):

$ PASSPHRASE=test duplicity --gpg-binary /usr/bin/gpg2 duplicity-0.7.10.tar.gz rsync://solarix::duplicity
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Sep 15 10:15:53 2016
--------------[ Backup Statistics ]--------------
StartTime 1473927433.08 (Thu Sep 15 10:17:13 2016)
EndTime 1473927433.09 (Thu Sep 15 10:17:13 2016)
ElapsedTime 0.00 (0.00 seconds)
SourceFiles 1
SourceFileSize 1543523 (1.47 MB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 0
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 102 (102 bytes)
Errors 0
-------------------------------------------------

$ ls -al /export/RsyncTest/duplicity-full*
-rw------- 1 predrag_zecevic admin  36K Sep 15 10:15 /export/RsyncTest/duplicity-full-signatures.20160915T081553Z.sigtar.gpg
-rw------- 1 predrag_zecevic admin  218 Sep 15 10:15 /export/RsyncTest/duplicity-full.20160915T081553Z.manifest.gpg
-rw------- 1 predrag_zecevic admin 1.5M Sep 15 10:15 /export/RsyncTest/duplicity-full.20160915T081553Z.vol1.difftar.gpg

#8 Updated by Predrag Zečević over 2 years ago

Aurélien,
I guess I can prepare component, so please specify it.
Thanks

#9 Updated by Aurélien Larcher over 2 years ago

Predrag,
thanks for the update.
As lockfile is a dependency it should be packaged as well.

You can prepare the components by copying this file:

https://github.com/OpenIndiana/oi-userland/blob/oi/hipster/templates/python.mk

You should fill-in the metada.

The components should be prepared for Python 2.7 at least.

Copy the same structure as:

https://github.com/OpenIndiana/oi-userland/tree/oi/hipster/components/python/elixir

If you want me to test and commit them, just send them to me by email.

Thanks!

Also available in: Atom