backup.rsannotatedbackup.rssource439 lines · 19.4 KB · raw

Backing up the device's data to one zip file, and putting it back.

What is in a backup is an allowlist, not "whatever is in the directory": the database (the chat, what Whiskers remembers, the parents' log and the copy of every device's log, the grown-ups' choices and the day's time, and every picture, all of them rows and blobs in the one file) as a consistent copy taken with SQLite's backup interface while the app runs, and the device's service address. The app's own diagnostics and the picture cache are not data and are never read here. The device id file is left out on purpose: it names ONE device in the day's time count and in nothing else, so a backup restored onto a different device (or onto the same one after a reinstall) must not make two devices claim the same name. A restored directory has no id, so the engine makes a fresh one at its next start, and the old id's minutes stay in the household document as another device's: the day's total keeps them, nothing is counted twice. The service address is device-local and is carried as one optional entry that the shell may offer to apply; it never goes into the data directory.

The zip begins with manifest.json (format version, when, which app, every entry with its size). Unpacking trusts nothing: the version must be one this build knows, every name must be on the allowlist (so .., absolute paths, backslashes, drive letters and links cannot be expressed), every entry must be in the manifest at exactly its size and no entry may be missing. It is all unpacked into a sibling directory first; only a complete, checked tree is swapped in, and the old data is set aside and removed only after the swap worked. A corrupt or truncated file therefore leaves the current data exactly as it was.

Errors say what happened to the grown-up in plain words and never quote anything from a file.

25use std::collections::{BTreeMap, BTreeSet};
26use std::fs::{self, File};
27use std::io::{self, Read, Write};
28use std::path::{Path, PathBuf};
29use std::time::{SystemTime, UNIX_EPOCH};
31use log::{error, info, warn};
32use serde::{Deserialize, Serialize};
33use zip::write::SimpleFileOptions;
34use zip::{CompressionMethod, ZipArchive, ZipWriter};

The only format this build writes and the only one it reads. Format 1 held the state as JSON files and a directory of pictures; this build has no reader for it, and says so ("a form this Whiskers does not know").

38pub const FORMAT_VERSION: u32 = 2;
39const MANIFEST: &str = "manifest.json";

The device-local service address, as UTF-8 text. Offered to the shell, never put in the data directory.

41pub const SERVICE_ADDRESS_ENTRY: &str = "device/service-address.txt";

The database, by its name in the data directory and in a backup.

43const DATABASE_ENTRY: &str = crate::DATABASE;

More than a household of a young child's pictures can come to; a manifest claiming more is refused before anything is written.

46const MAX_TOTAL_BYTES: u64 = 4 << 30;
47const MAX_ENTRIES: usize = 100_000;
48const MAX_ADDRESS_BYTES: u64 = 512;

Why a backup or a restore did not happen. Every message is a sentence for a grown-up.

51#[derive(Debug, PartialEq, Eq)]
52pub enum BackupError {

There is no data to back up yet.

54    NothingToBackUp,

The data on this device could not be read.

56    CannotRead,

The backup file could not be written (full storage, missing folder).

58    CannotWrite,

The file is not a Whiskers backup, or is damaged or cut short.

60    NotABackup,

Made by a newer Whiskers.

62    TooNew,

Made by a Whiskers this one does not know how to read.

64    UnknownVersion,

It holds something a backup never holds.

66    Unsafe,

It does not match its own list of contents.

68    Mismatch,

Larger than a backup can be.

70    TooLarge,

Checked fine but could not be put in place; what was on the device is untouched.

72    CannotInstall,
73}
75impl std::fmt::Display for BackupError {
76    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
77        f.write_str(match self {
78            Self::NothingToBackUp => "There is nothing to back up yet.",
79            Self::CannotRead => "Whiskers could not read its own data, so nothing was backed up.",
80            Self::CannotWrite => "The backup could not be saved. Check that the device has room, then try again.",
81            Self::NotABackup => "That file is not a Whiskers backup, or it is damaged. Nothing was changed.",
82            Self::TooNew => "That backup was made by a newer Whiskers. Update the app first. Nothing was changed.",
83            Self::UnknownVersion => "That backup is in a form this Whiskers does not know. Nothing was changed.",
84            Self::Unsafe => "That file holds things a Whiskers backup never holds, so it was not used. Nothing was changed.",
85            Self::Mismatch => "That backup does not match its own list of contents, so it was not used. Nothing was changed.",
86            Self::TooLarge => "That backup is far larger than a Whiskers backup can be, so it was not used. Nothing was changed.",
87            Self::CannotInstall => "The backup is fine but could not be put in place. What was on this device is untouched.",
88        })
89    }
90}
91
92impl std::error::Error for BackupError {}
93
94#[derive(Serialize, Deserialize, Debug)]
95struct Manifest {
96    format: u32,
97    created_at_ms: u64,
98    app_version: String,
99    entries: Vec<Entry>,
100}
101
102#[derive(Serialize, Deserialize, Debug, Clone)]
103struct Entry {
104    name: String,
105    size: u64,
106}

What a backup came to.

109#[derive(Debug, PartialEq, Eq)]
110pub struct BackedUp {
111    pub created_at_ms: u64,

State files (the database) and the pictures in it, not counting the service address.

113    pub files: u32,
114    pub pictures: u32,

Size of the zip file.

116    pub bytes: u64,
117}

What a restore brought back.

120#[derive(Debug, PartialEq, Eq)]
121pub struct Restored {
122    pub created_at_ms: u64,
123    pub app_version: String,
124    pub files: u32,
125    pub pictures: u32,

The service address the backup carried, in the core's own form, for the shell to offer; None if it carried none or one the core does not accept.

128    pub service_address: Option<String>,
129}

Whether an entry name may exist in a backup at all. The one gate: everything else follows from it.

132fn allowed(name: &str) -> bool {
133    name == DATABASE_ENTRY || name == SERVICE_ADDRESS_ENTRY
134}
136fn now_ms() -> u64 {
137    SystemTime::now().duration_since(UNIX_EPOCH).map_or(0, |d| d.as_millis() as u64)
138}
139
140fn options() -> SimpleFileOptions {
141    SimpleFileOptions::default().compression_method(CompressionMethod::Deflated)
142}

Writes a backup of data_dir to the file dest (replaced whole or not at all). service_address is carried as the optional device-local entry.

146pub fn backup_to(data_dir: &Path, dest: &Path, service_address: Option<&str>) -> Result<BackedUp, BackupError> {
147    info!("backup: starting");
148    let db = data_dir.join(DATABASE_ENTRY);
149    if !db.is_file() {
150        info!("backup: nothing to back up");
151        return Err(BackupError::NothingToBackUp);
152    }
153    // A consistent copy is made first, beside the destination, so the manifest's size is exactly what is
154    // written even while the app keeps writing to the database in use.
155    let copy = dest.with_file_name(format!("{}.db-copy", dest.file_name().map(|n| n.to_string_lossy().into_owned()).unwrap_or_default()));
156    // Made first, so that a destination that cannot be written to says so, and is not taken for unreadable data.
157    File::create(&copy).map_err(|e| {
158        error!("backup: cannot create the working copy ({:?})", e.kind());
159        BackupError::CannotWrite
160    })?;
161    let held = whiskers_store::copy_database(&db, &copy).map_err(|e| {
162        error!("backup: cannot copy the database ({e})");
163        let _ = fs::remove_file(&copy);
164        BackupError::CannotRead
165    })?;
166    let result = zip_up(&copy, dest, service_address, held.pictures);
167    let _ = fs::remove_file(&copy);
168    result
169}
171fn zip_up(copy: &Path, dest: &Path, service_address: Option<&str>, pictures: u32) -> Result<BackedUp, BackupError> {
172    let size = fs::metadata(copy).map_err(|_| BackupError::CannotRead)?.len();
173    let address = service_address.map(str::trim).filter(|a| !a.is_empty() && a.len() as u64 <= MAX_ADDRESS_BYTES);
174    let mut entries = vec![Entry { name: DATABASE_ENTRY.into(), size }];
175    if let Some(a) = address {
176        entries.push(Entry { name: SERVICE_ADDRESS_ENTRY.into(), size: a.len() as u64 });
177    }
178    let created_at_ms = now_ms();
179    let manifest = Manifest { format: FORMAT_VERSION, created_at_ms, app_version: env!("CARGO_PKG_VERSION").into(), entries };
180    let manifest_bytes = serde_json::to_vec_pretty(&manifest).map_err(|_| BackupError::CannotWrite)?;
181
182    let part = part_path(dest);
183    let result = (|| -> Result<u64, BackupError> {
184        let fail = |what: &'static str| move |e: &dyn std::fmt::Debug| {
185            error!("backup: {what} failed ({e:?})");
186            BackupError::CannotWrite
187        };
188        let file = File::create(&part).map_err(|e| fail("creating the file")(&e.kind()))?;
189        let mut zip = ZipWriter::new(file);
190        let w = |e: zip::result::ZipError| fail("writing the zip")(&e.to_string().len());
191        zip.start_file(MANIFEST, options()).map_err(w)?;
192        zip.write_all(&manifest_bytes).map_err(|e| fail("writing the manifest")(&e.kind()))?;
193        zip.start_file(DATABASE_ENTRY, options()).map_err(w)?;
194        let mut src = File::open(copy).map_err(|e| {
195            error!("backup: cannot open the database copy ({:?})", e.kind());
196            BackupError::CannotRead
197        })?;
198        let copied = io::copy(&mut Read::take(&mut src, size), &mut zip).map_err(|e| fail("copying the database")(&e.kind()))?;
199        if copied != size {
200            error!("backup: the database copy changed while it was zipped");
201            return Err(BackupError::CannotRead);
202        }
203        if let Some(a) = address {
204            zip.start_file(SERVICE_ADDRESS_ENTRY, options()).map_err(w)?;
205            zip.write_all(a.as_bytes()).map_err(|e| fail("writing the address")(&e.kind()))?;
206        }
207        let file = zip.finish().map_err(w)?;
208        file.sync_all().map_err(|e| fail("syncing")(&e.kind()))?;
209        let bytes = file.metadata().map_err(|e| fail("measuring")(&e.kind()))?.len();
210        fs::rename(&part, dest).map_err(|e| fail("moving into place")(&e.kind()))?;
211        Ok(bytes)
212    })();
213    match result {
214        Ok(bytes) => {
215            info!("backup: wrote the database ({size} bytes, {pictures} pictures), {bytes} bytes in all");
216            Ok(BackedUp { created_at_ms, files: 1, pictures, bytes })
217        }
218        Err(e) => {
219            let _ = fs::remove_file(&part);
220            Err(e)
221        }
222    }
223}
224
225fn part_path(dest: &Path) -> PathBuf {
226    let mut name = dest.file_name().map(|n| n.to_os_string()).unwrap_or_default();
227    name.push(".part");
228    dest.with_file_name(name)
229}

The two filesystem steps of the swap, so a test can make the second one fail.

232pub(crate) trait Rename {
233    fn rename(&self, from: &Path, to: &Path) -> io::Result<()>;
234}
236struct RealFs;
237impl Rename for RealFs {
238    fn rename(&self, from: &Path, to: &Path) -> io::Result<()> {
239        fs::rename(from, to)
240    }
241}

Replaces the contents of data_dir with the backup at src. The engine must not be running on data_dir: the shell stops it first and restarts the process after.

245pub fn restore_from(data_dir: &Path, src: &Path) -> Result<Restored, BackupError> {
246    restore_with(data_dir, src, &RealFs)
247}
249pub(crate) fn restore_with(data_dir: &Path, src: &Path, fs_ops: &dyn Rename) -> Result<Restored, BackupError> {
250    info!("restore: starting");
251    let parent = data_dir.parent().ok_or(BackupError::CannotInstall)?;
252    let stem = data_dir.file_name().map(|n| n.to_string_lossy().into_owned()).ok_or(BackupError::CannotInstall)?;
253    let stamp = now_ms();
254    let staging = parent.join(format!("{stem}.restoring-{stamp}"));
255    let aside = parent.join(format!("{stem}.replaced-{stamp}"));
256
257    let file = File::open(src).map_err(|e| {
258        error!("restore: cannot open the backup ({:?})", e.kind());
259        BackupError::NotABackup
260    })?;
261    let mut zip = ZipArchive::new(file).map_err(|_| {
262        warn!("restore: not a readable zip");
263        BackupError::NotABackup
264    })?;
265
266    let manifest = read_manifest(&mut zip)?;
267    let expected = check_manifest(&manifest)?;
268
269    // Everything is unpacked and checked before the data on the device is touched.
270    let unpacked = unpack(&mut zip, &expected, &staging);
271    let (files, service_address) = match unpacked {
272        Ok(v) => v,
273        Err(e) => {
274            let _ = fs::remove_dir_all(&staging);
275            return Err(e);
276        }
277    };
278    // What was unpacked must be a database this build can read, or nothing is touched.
279    let pictures = match check_database(&staging) {
280        Ok(n) => n,
281        Err(e) => {
282            let _ = fs::remove_dir_all(&staging);
283            return Err(e);
284        }
285    };
286
287    let had_old = data_dir.exists();
288    if had_old {
289        if let Err(e) = fs_ops.rename(data_dir, &aside) {
290            error!("restore: cannot set the current data aside ({:?})", e.kind());
291            let _ = fs::remove_dir_all(&staging);
292            return Err(BackupError::CannotInstall);
293        }
294    }
295    if let Err(e) = fs_ops.rename(&staging, data_dir) {
296        error!("restore: cannot move the restored data into place ({:?})", e.kind());
297        if had_old {
298            if let Err(back) = fs_ops.rename(&aside, data_dir) {
299                // The one state that must be loud: the old data is at `aside`, not at `data_dir`.
300                error!("restore: COULD NOT PUT THE OLD DATA BACK ({:?}); it is kept beside the data directory", back.kind());
301            }
302        }
303        let _ = fs::remove_dir_all(&staging);
304        return Err(BackupError::CannotInstall);
305    }
306    if had_old {
307        if let Err(e) = fs::remove_dir_all(&aside) {
308            warn!("restore: the replaced data could not be removed ({:?}); it is kept beside the data directory", e.kind());
309        }
310    }
311    // Only an address the core accepts is offered; anything else in that entry is dropped, not shown.
312    let service_address = service_address.and_then(|t| whiskers_core::ServiceAddress::parse(t.trim()).ok()).map(|a| a.text());
313    info!("restore: done, {files} state file(s), {pictures} pictures, address carried = {}", service_address.is_some());
314    Ok(Restored { created_at_ms: manifest.created_at_ms, app_version: manifest.app_version, files, pictures, service_address })
315}

Opens the unpacked database to see that it is one: a file that is not, or one from a newer Whiskers, is refused. Says how many pictures it holds.

319fn check_database(staging: &Path) -> Result<u32, BackupError> {
320    let db = staging.join(DATABASE_ENTRY);
321    let held = whiskers_store::inspect(&db).map_err(|e| {
322        warn!("restore: the database in the backup cannot be used ({e})");
323        if e.is_newer() { BackupError::TooNew } else { BackupError::NotABackup }
324    });
325    // Opening leaves no write-ahead log when it closes cleanly; anything left would not belong to the restored data.
326    for suffix in ["-wal", "-shm"] {
327        let _ = fs::remove_file(staging.join(format!("{DATABASE_ENTRY}{suffix}")));
328    }
329    held.map(|h| h.pictures)
330}
332fn read_manifest(zip: &mut ZipArchive<File>) -> Result<Manifest, BackupError> {
333    if zip.is_empty() {
334        return Err(BackupError::NotABackup);
335    }
336    let mut first = zip.by_index(0).map_err(|_| BackupError::NotABackup)?;
337    if first.name() != MANIFEST || first.is_dir() || first.size() > (4 << 20) {
338        return Err(BackupError::NotABackup);
339    }
340    let mut bytes = Vec::new();
341    first.by_ref().take(4 << 20).read_to_end(&mut bytes).map_err(|_| BackupError::NotABackup)?;
342    // The version is read before the rest, so a newer format's other changes cannot read as "damaged".
343    #[derive(Deserialize)]
344    struct Version {
345        format: u32,
346    }
347    let version: Version = serde_json::from_slice(&bytes).map_err(|_| BackupError::NotABackup)?;
348    match version.format {
349        FORMAT_VERSION => {}
350        0 => return Err(BackupError::NotABackup),
351        v if v > FORMAT_VERSION => return Err(BackupError::TooNew),
352        _ => return Err(BackupError::UnknownVersion),
353    }
354    serde_json::from_slice(&bytes).map_err(|_| BackupError::NotABackup)
355}

The manifest's entries by name, after every rule that needs only the manifest.

358fn check_manifest(m: &Manifest) -> Result<BTreeMap<String, u64>, BackupError> {
359    if m.entries.len() > MAX_ENTRIES {
360        return Err(BackupError::TooLarge);
361    }
362    let mut by_name = BTreeMap::new();
363    let mut total: u64 = 0;
364    for e in &m.entries {
365        if !allowed(&e.name) {
366            warn!("restore: the manifest lists a name a backup never holds");
367            return Err(BackupError::Unsafe);
368        }
369        if by_name.insert(e.name.clone(), e.size).is_some() {
370            return Err(BackupError::Mismatch);
371        }
372        total = total.saturating_add(e.size);
373    }
374    if total > MAX_TOTAL_BYTES {
375        return Err(BackupError::TooLarge);
376    }
377    if by_name.get(SERVICE_ADDRESS_ENTRY).is_some_and(|s| *s > MAX_ADDRESS_BYTES) {
378        return Err(BackupError::Unsafe);
379    }
380    Ok(by_name)
381}
383fn unpack(zip: &mut ZipArchive<File>, expected: &BTreeMap<String, u64>, staging: &Path) -> Result<(u32, Option<String>), BackupError> {
384    let cannot_stage = |e: io::Error| {
385        error!("restore: cannot write the staging directory ({:?})", e.kind());
386        BackupError::CannotInstall
387    };
388    fs::create_dir_all(staging).map_err(cannot_stage)?;
389    let mut seen: BTreeSet<String> = BTreeSet::new();
390    let mut files = 0u32;
391    let mut address = None;
392    if zip.len() != expected.len() + 1 {
393        return Err(BackupError::Mismatch);
394    }
395    for i in 1..zip.len() {
396        let mut entry = zip.by_index(i).map_err(|_| BackupError::NotABackup)?;
397        let name = entry.name().to_owned();
398        if entry.is_dir() || entry.is_symlink() || !allowed(&name) {
399            warn!("restore: an entry is a directory, a link or has a name a backup never holds");
400            return Err(BackupError::Unsafe);
401        }
402        let Some(&size) = expected.get(&name) else { return Err(BackupError::Mismatch) };
403        if !seen.insert(name.clone()) || entry.size() != size {
404            return Err(BackupError::Mismatch);
405        }
406        if name == SERVICE_ADDRESS_ENTRY {
407            let mut text = String::new();
408            entry.by_ref().take(size + 1).read_to_string(&mut text).map_err(|_| BackupError::NotABackup)?;
409            if text.len() as u64 != size {
410                return Err(BackupError::Mismatch);
411            }
412            address = Some(text);
413            continue;
414        }
415        let out_path = staging.join(&name);
416        let mut out = File::create(&out_path).map_err(cannot_stage)?;
417        // One byte past the declared size, so an entry that lies about its size is seen, not truncated.
418        let copied = io::copy(&mut entry.by_ref().take(size + 1), &mut out).map_err(|e| {
419            warn!("restore: an entry could not be read ({:?})", e.kind());
420            BackupError::NotABackup
421        })?;
422        if copied != size {
423            return Err(BackupError::Mismatch);
424        }
425        out.sync_all().map_err(cannot_stage)?;
426        files += 1;
427    }
428    if seen.len() != expected.len() {
429        return Err(BackupError::Mismatch);
430    }
431    if files == 0 {
432        // An address and nothing else is not a backup of anything.
433        return Err(BackupError::Mismatch);
434    }
435    Ok((files, address))
436}
437
438#[cfg(test)]
439mod tests;