Web Archive CLI
===============

Simple Python CLI to archive whole websites to the Web Archive via a `sitemap.xml` file.


Installation
------------

Create a fresh Python virtual env:

```
python3 -m venv venv
```

and activate it:

```
. venv/bin/activate
```

and install the dependencies:

```
pip install -r requirements.txt
```


Usage
-----

Activate the Python virtual env:

```
. venv/bin/activate
```

Convert a `sitemap.xml` file to a plain list of URLs:

```
python sitemap_to_urllist.py sitemap_example.org_2023-01-01.xml
```

Push all URLs to the web archive:

```
python do_archive.py urls_example.org_2023-01-01.txt
```

Note: Strictly follow the scheme with url and date encoded into the file names.


Dependencies
------------

The archive script is based on the `savepagenow` Python package:

* https://pypi.org/project/savepagenow/
* https://github.com/palewire/savepagenow


To archive a single URL only, the `savepagenow` CLI can be used directly:

* https://palewi.re/docs/savepagenow/cli.html


Links
-----

Wayback API:

* https://archive.org/help/wayback_api.php

Manual paper feed:

* https://web.archive.org/save/