Project

General

Profile

Actions
Copy link

Feature #7157

closed

Add duplicity (Encrypted bandwidth-efficient backup using the rsync algorithm)

Added by Predrag Zečević about 7 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Predrag Zečević
Category:
OI-Userland
Target version:
Hipster
Start date:
2016-07-01
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:

Description

Hi,

I have tried to get duplicity (http://duplicity.nongnu.org/index.html) working on /hipster:
  1. Had to install prerequisite package:
    pfexec pkg install -v library/librsync
  2. Got latest duplicity downloaded:
    wget https://code.launchpad.net/duplicity/0.7-series/0.7.07.1/+download/duplicity-0.7.07.1.tar.gz
tar xf duplicity-0.7.07.1.tar.gz
cd duplicity-0.7.07.1
pfexec /usr/bin/python2.7 setup.py install --prefix=/opt/SFW
  3. Fixed python call (to use python2.7):
    pfexec vim /opt/SFW/bin/duplicity
  4. Calling produces error:
    $ duplicity -V
Traceback (most recent call last):
  File "/opt/SFW/bin/duplicity", line 45, in <module>
    from lockfile import LockFile as FileLock
ImportError: No module named lockfile

    Looked after that error on google and in most cases some "python-lockfile", or "py27-lockfile", etc package was fix.
    Searched around and could not find it in OI (/hipster). Is it complicated to add one (e.g. python lockfile)?
    From the other side, https://pypi.python.org/pypi/lockfile suggests that it is obsoleted and one has to use https://pypi.python.org/pypi/fasteners or http://docs.openstack.org/developer/oslo.concurrency/
    • Fastener is already there:
      $ pfexec /usr/bin/pip-2.7 install fasteners
Requirement already satisfied (use --upgrade to upgrade): fasteners in /usr/lib/python2.7/site-packages
    • Installing oslo.concurenty is as simple as:
      pfexec /usr/bin/pip-2.7 install oslo.concurrency
    • Even tried to add lockfile:
      $ pfexec /usr/bin/pip-2.7 install lockfile

      but neither does help... Somehow I am lost here.

BTW, tried also with default python - no changes...

Maybe someone else knows how to get this done properly?

Regards.

Actions
Copy link
 #1

Updated by Predrag Zečević about 7 years ago

UPDATE:

I have created backup BE and installed duplicity (and fixed python call) to system regular path with:

pfexec /usr/bin/pip install lockfile
pfexec /usr/bin/python setup.py install 
pfexec vim /usr/bin/duplicity # fix python call: python2 --> python2.6

And now it does not complain.... I need to to check if it is working properly.
$ /usr/bin/duplicity -V
duplicity 0.7.07.1

Would be nice then, to have lockfile installed before adding duplicity.

I will update ticket once I do some tests.

Regards.

Actions
Copy link
 #2

Updated by Aurélien Larcher about 7 years ago

  • Category set to OI-Userland
  • Assignee set to Predrag Zečević
  • Target version set to Hipster
  • Tags deleted (needs-triage)
Actions
Copy link
 #3

Updated by Aurélien Larcher about 7 years ago

Any update? Can you prepare the component?

Actions
Copy link
 #4

Updated by Predrag Zečević about 7 years ago

Will update soon

Actions
Copy link
 #5

Updated by Predrag Zečević about 7 years ago

There is new version available, so tested in zone:

  • Install prerequisites:
    $ pfexec pkg install -v library/librsync network/ftp/lftp crypto/gnupg
: # ...

$ pfexec /usr/bin/pip install lockfile
Downloading/unpacking lockfile
  Downloading lockfile-0.12.2.tar.gz
  Running setup.py egg_info for package lockfile

    Installed /tmp/pip_build_root/lockfile/.eggs/pbr-1.10.0-py2.7.egg
    [pbr] Processing SOURCES.txt
    warning: LocalManifestMaker: standard file '-c' not found

    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Installing collected packages: lockfile
  Running setup.py install for lockfile
    [pbr] Generating AUTHORS
    [pbr] AUTHORS complete (0.0s)
    [pbr] Reusing existing SOURCES.txt
Successfully installed lockfile
Cleaning up...

$ which python
/usr/bin/python

$ python --version
Python 2.7.12
  • Get source
    $ wget https://code.launchpad.net/duplicity/0.7-series/0.7.10/+download/duplicity-0.7.10.tar.gz
$ tar xf duplicity-0.7.10.tar.gz
$ cd duplicity-0.7.10
  • Install:
    $ pfexec python setup.py install
running install
running build
running build_py
creating build/lib.solaris-2.11-i86pc.32bit-2.7/testing
copying testing/__init__.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing
copying testing/test_code.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing
creating build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
copying testing/functional/test_cleanup.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
copying testing/functional/test_log.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
copying testing/functional/test_restart.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
copying testing/functional/test_verify.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
copying testing/functional/test_selection.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
copying testing/functional/test_final.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
copying testing/functional/__init__.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
copying testing/functional/test_rdiffdir.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
copying testing/functional/test_badupload.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/functional
creating build/lib.solaris-2.11-i86pc.32bit-2.7/testing/overrides
copying testing/overrides/gettext.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/overrides
copying testing/overrides/__init__.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/overrides
creating build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_lazy.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_gpginterface.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_file_naming.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_backend_instance.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_statistics.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_manifest.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_collections.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_dup_temp.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_dup_time.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_gpg.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_backend.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_globmatch.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/__init__.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_tarfile.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_tempdir.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_path.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_patchdir.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_selection.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
copying testing/unit/test_diffdir.py -> build/lib.solaris-2.11-i86pc.32bit-2.7/testing/unit
running build_ext
running build_scripts
running install_lib
running install_data
running install_egg_info
running egg_info
writing requirements to duplicity.egg-info/requires.txt
writing duplicity.egg-info/PKG-INFO
writing top-level names to duplicity.egg-info/top_level.txt
writing dependency_links to duplicity.egg-info/dependency_links.txt
reading manifest file 'duplicity.egg-info/SOURCES.txt'
writing manifest file 'duplicity.egg-info/SOURCES.txt'
removing '/usr/lib/python2.7/site-packages/duplicity-0.7.10-py2.7.egg-info' (and everything under it)
Copying duplicity.egg-info to /usr/lib/python2.7/site-packages/duplicity-0.7.10-py2.7.egg-info
running install_scripts
changing mode of /usr/bin/rdiffdir to 755
changing mode of /usr/bin/duplicity to 755
  • Fix python call:
    $ pfexec sed -i 's#env python2#env python#' /usr/bin/duplicity
$ duplicity --version
duplicity 0.7.10

$ man duplicity
cat: /usr/share/man/man1/duplicity.1: Permission denied
()                                                                          ()

                              September 15, 2016                            ()
$ ls -al /usr/share/man/man1/duplicity.1
-rwx------ 1 root root 65611 Aug 20 21:13 /usr/share/man/man1/duplicity.1
$ pfexec chmod 664 /usr/share/man/man1/duplicity.1

    Will do test backup and paste findings here
Actions
Copy link
 #6

Updated by Predrag Zečević about 7 years ago

Test with ftp sync (it expects 'gpg' binary, not 'gpg2' as we have):

$ FTP_PASSWORD=my@ftp duplicity --gpg-binary /usr/bin/gpg2 duplicity-0.7.10.tar.gz ftp://ftp@solarix:/incoming
LFTP version is 4.3.8
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
GnuPG passphrase: 
Retype passphrase to confirm: 
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1473926013.01 (Thu Sep 15 09:53:33 2016)
EndTime 1473926013.59 (Thu Sep 15 09:53:33 2016)
ElapsedTime 0.58 (0.58 seconds)
SourceFiles 1
SourceFileSize 1543523 (1.47 MB)
NewFiles 1
NewFileSize 1543523 (1.47 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1
RawDeltaSize 1543523 (1.47 MB)
TotalDestinationSizeChange 1546442 (1.47 MB)
Errors 0
-------------------------------------------------

Check for files:
$ ls -al /export/Ftp/incoming/duplicity-full*
-rw-r--r-- 1 ftp nogroup  36K Sep 15 09:53 /export/Ftp/incoming/duplicity-full-signatures.20160915T075329Z.sigtar.gpg
-rw-r--r-- 1 ftp nogroup  217 Sep 15 09:53 /export/Ftp/incoming/duplicity-full.20160915T075329Z.manifest.gpg
-rw-r--r-- 1 ftp nogroup 1.5M Sep 15 09:53 /export/Ftp/incoming/duplicity-full.20160915T075329Z.vol1.difftar.gpg

Actions
Copy link
 #7

Updated by Predrag Zečević about 7 years ago

Test with rsync (only to rsync server - so no user/password needed):

$ PASSPHRASE=test duplicity --gpg-binary /usr/bin/gpg2 duplicity-0.7.10.tar.gz rsync://solarix::duplicity
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Sep 15 10:15:53 2016
--------------[ Backup Statistics ]--------------
StartTime 1473927433.08 (Thu Sep 15 10:17:13 2016)
EndTime 1473927433.09 (Thu Sep 15 10:17:13 2016)
ElapsedTime 0.00 (0.00 seconds)
SourceFiles 1
SourceFileSize 1543523 (1.47 MB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 0
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 102 (102 bytes)
Errors 0
-------------------------------------------------

$ ls -al /export/RsyncTest/duplicity-full*
-rw------- 1 predrag_zecevic admin  36K Sep 15 10:15 /export/RsyncTest/duplicity-full-signatures.20160915T081553Z.sigtar.gpg
-rw------- 1 predrag_zecevic admin  218 Sep 15 10:15 /export/RsyncTest/duplicity-full.20160915T081553Z.manifest.gpg
-rw------- 1 predrag_zecevic admin 1.5M Sep 15 10:15 /export/RsyncTest/duplicity-full.20160915T081553Z.vol1.difftar.gpg

Actions
Copy link
 #8

Updated by Predrag Zečević about 7 years ago

Aurélien,
I guess I can prepare component, so please specify it.
Thanks

Actions
Copy link
 #9

Updated by Aurélien Larcher about 7 years ago

Predrag,
thanks for the update.
As lockfile is a dependency it should be packaged as well.

You can prepare the components by copying this file:

https://github.com/OpenIndiana/oi-userland/blob/oi/hipster/templates/python.mk

You should fill-in the metada.

The components should be prepared for Python 2.7 at least.

Copy the same structure as:

https://github.com/OpenIndiana/oi-userland/tree/oi/hipster/components/python/elixir

If you want me to test and commit them, just send them to me by email.

Thanks!

Actions
Copy link
 #10

Updated by Predrag Zečević about 4 years ago

I would like to close this (will use restic instead - they even supply Solaris version) - but I do not see how.

Regards.

Actions
Copy link
 #11

Updated by Michal Nowak about 4 years ago

  • Status changed from New to Closed
Actions
Copy link

Also available in: Atom PDF