aner/homey
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

TMP COMMIT BEFORE TRASHING

Browse Source
This commit is contained in:
Aner Zakobar
2026-04-15 16:49:18 +03:00
parent 138d6d8a6b
commit d1948df47e
18 changed files with 1389 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+1
View File
@@ -1,2 +1,3 @@
charts charts
*.lock *.lock
.agent-shell
AGENTS.md
+255
View File
@@ -0,0 +1,255 @@
# AGENTS.md
This is a Helm chart for deploying a self-hosted home environment (Homey) on Kubernetes.
## Project Overview
- **Type**: Helm Chart (Kubernetes package manager)
- **Language**: YAML + Go template syntax (Helm templating)
- **Key Files**:
- `Chart.yaml` - Chart metadata
- `values.yaml` - Default configuration values
- `templates/` - Kubernetes manifest templates (auth.yaml, media.yaml, phpldapadmin.yaml, _definitions.yaml)
- `files/` - Configuration file templates (processed by Helm with `tpl` function)
## Build/Lint/Test Commands
### Helm Validation
```bash
# Lint the Helm chart for errors
helm lint .
# Template rendering (dry-run install)
helm template test-release . --debug
# Install/upgrade in cluster
helm upgrade --install homey . -n homey
# Verify chart against Kubernetes API
helm kubeval .
# Check schema validation of values.yaml
helm schema generate
```
### Manual Template Testing
```bash
# Render templates locally with custom values
helm template homey . -f values.yaml --set homey.url=example.com
# Template with debug output
helm template homey . --debug 2>&1 | less
```
### Kubectl Validation
```bash
# Dry-run apply to validate manifests
kubectl apply -f templates/auth.yaml --dry-run=server
# Get rendered template directly
helm template homey . | kubectl apply --dry-run=server -f -
```
## Code Style Guidelines
### YAML Structure
1. **Document Separators**: Use `---` at the start of each YAML document
```yaml
---
apiVersion: v1
kind: ConfigMap
```
2. **Indentation**: Use 2 spaces (not tabs)
```yaml
spec:
containers:
- name: app
image: nginx
```
3. **Trailing Commas**: Optional but preferred for multi-line lists
```yaml
accessModes:
- ReadWriteMany
- ReadOnlyMany
```
4. **Quotes**: Use quotes for strings that might be interpreted as other types
- Always quote: `.Values.homey.url | quote`
- Optional for simple strings like names
### Kubernetes Resources
1. **Labels**: Use Kubernetes recommended labels
```yaml
labels:
app.kubernetes.io/name: openldap
app.kubernetes.io/component: auth
```
2. **Naming**: Use kebab-case for resource names
```yaml
name: openldap-admin
name: nextcloud-postgres
```
3. **Storage**: Always specify `storageClassName: longhorn`
```yaml
spec:
storageClassName: longhorn
```
### Helm Template Syntax
1. **Variable Assignment**: Use `$_ := set` for complex assignments
```yaml
{{- $_ := set $ "varname" (include "homey.lookuporgensecret" (merge (dict "secretname" "secret-name") $)) }}
```
2. **Include with Merge**: Always pass `$` as the last argument
```yaml
{{ include "homey.randomsecret" (merge (dict "secretname" "secret-name" "secretval" $secretval) $) }}
```
3. **Quote Values from .Values**: Use `quote` filter
```yaml
value: {{ .Values.homey.url | quote }}
```
4. **Template Definitions**: Define reusable templates in `_definitions.yaml`
- `homey.lookuporgensecret` - Look up existing secrets or generate random
- `homey.randomsecret` - Generate a random secret
- `homey.randHex` - Generate random hex string
5. **Template Spacing**: Use whitespace control to avoid extra newlines
```yaml
{{- "leading minus" -}} # No newline before
{{ "trailing minus" -}} # No newline after
```
### Secret Management
1. **Annotations**: Always annotate managed secrets to prevent deletion
```yaml
annotations:
"helm.sh/resource-policy": "keep"
```
2. **Secret Generation Pattern**:
```yaml
# Check for existing secret, create if not exists
{{- $secretObj := (lookup "v1" "Secret" .Release.Namespace "secret-name") | default dict -}}
{{- $secretData := (get $secretObj "data") | default dict -}}
{{- $pass := (get $secretData "password") | default (randAlphaNum 32 | b64enc) -}}
```
3. **Never hardcode secrets** - Use the secret lookup pattern above
### Config Files (files/ directory)
1. **Go Templates in Configs**: Use `tpl` function to process config files
```yaml
data:
config.yml: |-
{{ tpl (.Files.Get "files/authelia-config.yaml" | indent 4) . }}
```
2. **Accessing Variables**: Config files can access `.Values.*` and custom variables set in templates
### Ingress Configuration
1. **TLS**: Always specify TLS with proper hosts and secret
```yaml
spec:
ingressClassName: {{ .Values.homey.ingress_class }}
tls:
- hosts:
- auth.{{ .Values.homey.url }}
secretName: {{ .Values.homey.certname }}
```
2. **Authelia Integration**: Use auth snippets for protected ingresses
```yaml
annotations:
nginx.ingress.kubernetes.io/auth-url: http://authelia.{{ .Release.Namespace }}.svc.cluster.local:9091/api/verify
nginx.ingress.kubernetes.io/auth-signin: https://auth.{{ .Values.homey.url }}?rm=$request_method
```
### Resource Organization
1. **File Structure**:
- `templates/_definitions.yaml` - Helper templates (secrets, utilities)
- `templates/auth.yaml` - Authentication services (OpenLDAP, Authelia, Gitea, Nextcloud, Radicale)
- `templates/media.yaml` - Media services (Jellyfin, Transmission)
- `templates/phpldapadmin.yaml` - LDAP admin interface
2. **Manifest Order** (within a file):
- PersistentVolumeClaim
- Secrets
- ConfigMaps
- Deployments
- Services
- Ingress
3. **Unused Resources**: Keep deprecated manifests in `unused/` directory
### Environment Variables
1. **Naming**: Use uppercase with underscores
```yaml
- name: LDAP_ORGANISATION
value: {{ .Values.homey.organization }}
```
2. **Value Sources**: Prefer `valueFrom.secretKeyRef` over inline values
```yaml
- name: PASSWORD
valueFrom:
secretKeyRef:
name: secret-name
key: password
```
### Volume Mounts
1. **subPath**: Use `subPath` for shared PVCs
```yaml
volumeMounts:
- mountPath: /data
subPath: service-name/data
```
2. **Read-only ConfigMaps**: Mark config mounts as read-only
```yaml
readOnly: true
```
## Common Operations
### Adding a New Service
1. Add values to `values.yaml`
2. Create/extend template in `templates/`
3. Add PVC if persistent storage needed
4. Add Ingress with appropriate annotations
5. Test with `helm template .`
### Updating Secrets
Secrets are generated on first install. To regenerate:
```bash
kubectl delete secret <secret-name> -n homey
helm upgrade --install homey . -n homey
```
### Debugging Templates
```bash
# Show all template variables available
helm template . --show-only templates/_helpers.tpl
# Render single template
helm template . --show-only templates/auth.yaml
```
backup-longhorn-volumes.sh
Executable
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="$HOME/homey-backup"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
NODE="root@192.168.1.100"
NC_DATA_PVC="pvc-5c1f48e3-346f-4c35-8e6a-8fc0c4c3a842-96d72815"
NC_DB_PVC="pvc-c5b28179-1b9c-462a-be5b-05c4f0bb36ca-5f2dbf4d"
NC_DATA_DEST="$BACKUP_DIR/longhorn-nextcloud-data/$TIMESTAMP"
NC_DB_DEST="$BACKUP_DIR/longhorn-nextcloud-db/$TIMESTAMP"
echo "=== Longhorn Volume Backup (Emergency) ==="
echo "Started: $(date)"
echo ""
echo "WARNING: Backing up raw Longhorn volume images"
echo "These are sparse files - actual data is smaller than file size"
echo ""
mkdir -p "$NC_DATA_DEST"
mkdir -p "$NC_DB_DEST"
echo "--- Backing up Nextcloud data volume ---"
echo "Source: $NODE:/hda/replicas/$NC_DATA_PVC/"
echo "Dest: $NC_DATA_DEST/"
echo ""
rsync -avzP --no-owner --no-group --sparse \
"$NODE:/hda/replicas/$NC_DATA_PVC/" \
"$NC_DATA_DEST/"
echo ""
echo "Nextcloud volume backup: $(du -sh "$NC_DATA_DEST" | cut -f1)"
echo ""
echo "--- Backing up PostgreSQL volume ---"
echo "Source: $NODE:/hda/replicas/$NC_DB_PVC/"
echo "Dest: $NC_DB_DEST/"
echo ""
rsync -avzP --no-owner --no-group --sparse \
"$NODE:/hda/replicas/$NC_DB_PVC/" \
"$NC_DB_DEST/"
echo ""
echo "PostgreSQL volume backup: $(du -sh "$NC_DB_DEST" | cut -f1)"
echo ""
echo "=== Backup Complete ==="
echo "Timestamp: $TIMESTAMP"
echo "Finished: $(date)"
echo ""
echo "Total backup size:"
du -sh "$BACKUP_DIR/longhorn-nextcloud-data/$TIMESTAMP" "$BACKUP_DIR/longhorn-nextcloud-db/$TIMESTAMP"
echo ""
echo "NOTE: These are raw Longhorn volume images."
echo "To restore, copy back to /hda/replicas/ and restart Longhorn."
backup-nextcloud.sh
Executable
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="$HOME/homey-backup"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
NAMESPACE="homey"
NODE="root@192.168.1.100"
NC_DATA_DEST="$BACKUP_DIR/nextcloud-data/backups/$TIMESTAMP"
NC_DB_DEST="$BACKUP_DIR/nextcloud-postgres/backups/$TIMESTAMP"
show_progress() {
local file="$1"
local label="$2"
local total="${3:-0}"
while [[ ! -f "$file" ]]; do
sleep 0.2
done
while kill -0 "$BACKUP_PID" 2>/dev/null; do
local size=$(stat -c%s "$file" 2>/dev/null || echo "0")
if [[ "$total" -gt 0 ]]; then
local pct=$((size * 100 / total))
local size_hr=$(numfmt --to=iec-i --suffix=B "$size" 2>/dev/null || echo "${size}B")
local total_hr=$(numfmt --to=iec-i --suffix=B "$total" 2>/dev/null || echo "${total}B")
printf "\r%s: %s / %s (%d%%) " "$label" "$size_hr" "$total_hr" "$pct"
else
local size_hr=$(numfmt --to=iec-i --suffix=B "$size" 2>/dev/null || echo "${size}B")
printf "\r%s: %s " "$label" "$size_hr"
fi
sleep 0.5
done
local final=$(stat -c%s "$file" 2>/dev/null || echo "0")
local final_hr=$(numfmt --to=iec-i --suffix=B "$final" 2>/dev/null || echo "${final}B")
printf "\r%s: %s - done! \n" "$label" "$final_hr"
}
wait_for_cluster() {
echo "Waiting for Kubernetes cluster..."
local max_wait=300
local start=$(date +%s)
while true; do
if kubectl get nodes &>/dev/null; then
echo "Cluster ready!"
return 0
fi
local now=$(date +%s)
if [[ $((now - start)) -ge $max_wait ]]; then
echo "ERROR: Cluster not available after ${max_wait}s"
return 1
fi
printf "\rWaiting... %ds " $((now - start))
sleep 5
done
}
echo "=== Nextcloud Backup ==="
echo "Started: $(date)"
mkdir -p "$NC_DATA_DEST"
mkdir -p "$NC_DB_DEST"
if ! wait_for_cluster; then
echo "Cluster unavailable. Cannot proceed."
exit 1
fi
echo ""
echo "--- Backing up PostgreSQL ---"
DB_POD=$(kubectl get pods -n "$NAMESPACE" -l app=nextcloud-postgres -o jsonpath='{.items[0].metadata.name}')
if [[ -z "$DB_POD" ]]; then
echo "ERROR: PostgreSQL pod not found"
exit 1
fi
DB_SIZE=$(kubectl exec -n "$NAMESPACE" "$DB_POD" -- psql -U postgres -d nextcloud_db -t -c "SELECT pg_database_size('nextcloud_db');" 2>/dev/null | tr -d ' ' || echo "0")
echo "Database size: $(numfmt --to=iec-i --suffix=B "$DB_SIZE" 2>/dev/null || echo "$DB_SIZE bytes")"
echo "Dumping database from $DB_POD..."
kubectl exec -n "$NAMESPACE" "$DB_POD" -- sh -c "pg_dump -U postgres nextcloud_db" | gzip > "$NC_DB_DEST/dump.sql.gz" &
BACKUP_PID=$!
show_progress "$NC_DB_DEST/dump.sql.gz" "Database" "$DB_SIZE"
wait $BACKUP_PID 2>/dev/null || true
echo "Database backup: $(du -sh "$NC_DB_DEST/dump.sql.gz" | cut -f1)"
echo ""
echo "--- Backing up Nextcloud data ---"
NC_POD=$(kubectl get pods -n "$NAMESPACE" -l app=nextcloud -o jsonpath='{.items[0].metadata.name}')
if [[ -z "$NC_POD" ]]; then
echo "ERROR: Nextcloud pod not found"
exit 1
fi
NC_SIZE=$(kubectl exec -n "$NAMESPACE" "$NC_POD" -- du -sb /var/www/html 2>/dev/null | awk '{print $1}' || echo "0")
echo "Data size: $(numfmt --to=iec-i --suffix=B "$NC_SIZE" 2>/dev/null || echo "$NC_SIZE bytes")"
echo "Creating tar archive from $NC_POD..."
kubectl exec -n "$NAMESPACE" "$NC_POD" -- tar cf - -C /var/www/html . 2>/dev/null | gzip > "$NC_DATA_DEST/data.tar.gz" &
BACKUP_PID=$!
show_progress "$NC_DATA_DEST/data.tar.gz" "Data" "$NC_SIZE"
wait $BACKUP_PID 2>/dev/null || true
echo "Data backup: $(du -sh "$NC_DATA_DEST/data.tar.gz" | cut -f1)"
echo ""
echo "--- Creating latest symlinks ---"
rm -f "$BACKUP_DIR/nextcloud-data/latest" "$BACKUP_DIR/nextcloud-postgres/latest"
ln -sf "backups/$TIMESTAMP" "$BACKUP_DIR/nextcloud-data/latest"
ln -sf "backups/$TIMESTAMP" "$BACKUP_DIR/nextcloud-postgres/latest"
echo ""
echo "=== Backup Complete ==="
echo "Timestamp: $TIMESTAMP"
echo "Finished: $(date)"
echo ""
du -sh "$NC_DATA_DEST" "$NC_DB_DEST"
copy-longhorn-from-hd.sh
Executable
+41
View File
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
set -euo pipefail
SRC="/mnt/replicas"
DEST="$HOME/homey-backup/longhorn-volumes"
SKIP="pvc-dfe2aa08-bbb8-423b-9001-fb6aea181597-baf06a7f"
mkdir -p "$DEST"
echo "=== Copying Longhorn volumes from HD ==="
echo "Source: $SRC"
echo "Dest: $DEST"
echo "Skip: $SKIP (Jellyfin)"
echo ""
for pvc in "$SRC"/*/; do
name=$(basename "$pvc")
if [[ "$name" == "$SKIP" ]]; then
echo "Skipping: $name"
continue
fi
echo ""
echo "Copying: $name"
src_size=$(sudo du -sb "$pvc" 2>/dev/null | awk '{print $1}' || echo "0")
src_size_hr=$(numfmt --to=iec-i --suffix=B "$src_size" 2>/dev/null || echo "${src_size}B")
echo "Size: $src_size_hr"
sudo rsync -a --no-owner --no-group --info=progress2 "${pvc%/}" "$DEST/"
sudo chown -R "$USER" "$DEST/$name"
size=$(du -sh "$DEST/$name" | cut -f1)
echo "Done: $size"
done
echo ""
echo "=== Copy Complete ==="
echo "Total size:"
sudo du -sh "$DEST"
scripts/backup-longhorn-to-disk.sh
Executable
+184
View File
@@ -0,0 +1,184 @@
#!/usr/bin/env bash
set -euo pipefail
SRC="${SRC:-/mnt/replicas}"
DEST="${DEST:-/mnt2/homey-backup}"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
MANIFEST="$DEST/manifest.json"
PVC_MAPPING=(
"pvc-0310a337-9642-464b-a458-fcb3439328e7-fbc07d5a:ldap-pvc"
"pvc-1cdc51ee-b965-4cab-baf7-077cc6df6f11-0fcfb9cd:authelia-pvc"
"pvc-4888bf84-62c8-4340-adbc-cb31073d8fd2-d065d20b:gitea-pvc"
"pvc-5c1f48e3-346f-4c35-8e6a-8fc0c4c3a842-96d72815:nextcloud-data-pvc"
"pvc-c5b28179-1b9c-462a-be5b-05c4f0bb36ca-5f2dbf4d:nextcloud-postgres-pvc"
"pvc-7f73ee94-5583-4e4a-9788-cba054214b1c-f767850a:radicale-pvc"
"pvc-9e75f35a-27c3-4251-b25a-1a876f82f6c7-c9c8b185:jellyfin-config-pvc"
"pvc-dfe2aa08-bbb8-423b-9001-fb6aea181597-baf06a7f:jellyfin-data-pvc"
"pvc-dd4a069a-a638-49c0-8c95-f954510816e5-7e81a6f6:transmission-config-pvc"
"pvc-e4ba414d-d9c2-4927-b0ae-f6bfb90ce311-a0963101:unknown-pvc-1"
"pvc-ec6afe10-aca3-42ce-9d89-32fc4ac77f9a-8d6baa34:unknown-pvc-2"
)
progress_bar() {
local current=$1
local total=$2
local width=40
local percent=$((current * 100 / total))
local filled=$((current * width / total))
local empty=$((width - filled))
printf "\r["
printf "%${filled}s" | tr ' ' '='
printf "%${empty}s" | tr ' ' ' '
printf "] %3d%% (%d/%d)" "$percent" "$current" "$total"
}
get_pvc_name() {
local pvc_id="$1"
for mapping in "${PVC_MAPPING[@]}"; do
if [[ "$mapping" == "$pvc_id:"* ]]; then
echo "${mapping#*:}"
return
fi
done
echo "unknown"
}
echo "========================================"
echo " Longhorn Volume Backup Tool"
echo "========================================"
echo ""
echo "Source: $SRC"
echo "Destination: $DEST"
echo "Timestamp: $TIMESTAMP"
echo ""
mkdir -p "$DEST/volumes"
mkdir -p "$DEST/metadata"
VOLUMES=()
TOTAL_SIZE=0
echo "Scanning volumes..."
for pvc_dir in "$SRC"/*/; do
pvc_name=$(basename "$pvc_dir")
friendly_name=$(get_pvc_name "$pvc_name")
VOLUMES+=("$pvc_name:$friendly_name")
size=$(sudo du -sb "$pvc_dir" 2>/dev/null | awk '{print $1}' || echo "0")
TOTAL_SIZE=$((TOTAL_SIZE + size))
printf " %-50s %s\n" "$friendly_name" "$(numfmt --to=iec-i --suffix=B "$size" 2>/dev/null || echo "${size}B")"
done
TOTAL_VOLUMES=${#VOLUMES[@]}
echo ""
echo "Found $TOTAL_VOLUMES volumes, total size: $(numfmt --to=iec-i --suffix=B "$TOTAL_SIZE")"
echo ""
read -p "Continue with backup? [y/N] " -n 1 -r
echo ""
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Aborted."
exit 1
fi
echo ""
echo "Starting backup..."
echo ""
COPIED_SIZE=0
START_TIME=$(date +%s)
for i in "${!VOLUMES[@]}"; do
volume="${VOLUMES[$i]}"
pvc_name="${volume%%:*}"
friendly_name="${volume#*:}"
CURRENT=$((i + 1))
progress_bar "$CURRENT" "$TOTAL_VOLUMES"
echo " - $friendly_name"
sudo rsync -a --no-owner --no-group --info=progress2 \
"$SRC/$pvc_name/" \
"$DEST/volumes/$pvc_name/" 2>&1 | while read -r line; do
if [[ "$line" =~ to-chk=*([0-9]+)/([0-9]+) ]]; then
printf "\r %s" "$line"
fi
done
sudo chown -R "$USER:$USER" "$DEST/volumes/$pvc_name" 2>/dev/null || true
if [[ -f "$SRC/$pvc_name/volume.meta" ]]; then
sudo cp "$SRC/$pvc_name/volume.meta" "$DEST/metadata/${pvc_name}.meta" 2>/dev/null || true
fi
echo ""
done
echo ""
echo "Generating manifest..."
cat > "$MANIFEST" << EOF
{
"backup_timestamp": "$TIMESTAMP",
"source_path": "$SRC",
"destination_path": "$DEST",
"total_volumes": $TOTAL_VOLUMES,
"total_size_bytes": $TOTAL_SIZE,
"volumes": [
EOF
FIRST=true
for volume in "${VOLUMES[@]}"; do
pvc_name="${volume%%:*}"
friendly_name="${volume#*:}"
vol_size=$(sudo du -sb "$SRC/$pvc_name" 2>/dev/null | awk '{print $1}' || echo "0")
vol_size_hr=$(numfmt --to=iec-i --suffix=B "$vol_size" 2>/dev/null || echo "${vol_size}B")
head_file=$(sudo find "$DEST/volumes/$pvc_name" -name "volume-head-*.img" 2>/dev/null | head -1)
head_file=$(basename "$head_file" 2>/dev/null || echo "")
if [[ "$FIRST" == "true" ]]; then
FIRST=false
else
echo "," >> "$MANIFEST"
fi
cat >> "$MANIFEST" << EOF
{
"pvc_id": "$pvc_name",
"friendly_name": "$friendly_name",
"size_bytes": $vol_size,
"size_human": "$vol_size_hr",
"volume_head": "$head_file",
"backup_path": "volumes/$pvc_name"
}
EOF
done
cat >> "$MANIFEST" << EOF
]
}
EOF
END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))
echo ""
echo "========================================"
echo " Backup Complete!"
echo "========================================"
echo ""
echo "Duration: $((DURATION / 60))m $((DURATION % 60))s"
echo "Location: $DEST"
echo "Manifest: $MANIFEST"
echo ""
echo "Backup size:"
sudo du -sh "$DEST/volumes"
echo ""
echo "To mount a volume, run:"
echo " ./scripts/mount-longhorn-volume.sh <pvc-name-or-friendly-name>"
echo ""
echo "To restore a volume, run:"
echo " ./scripts/restore-longhorn-volume.sh <pvc-name-or-friendly-name>"
scripts/get-pvc-mapping.sh
+16
View File
@@ -0,0 +1,16 @@
for dir in /mnt/replicas/pvc-*/; do
name=$(basename "$dir")
head=$(sudo find "$dir" -name "volume-head-*.img" | head -1)
sudo mkdir -p /tmp/inspect
loop=$(sudo losetup -fP --show "$head")
echo "=== $name ==="
sudo mount "$loop" /tmp/inspect 2>/dev/null && {
sudo ls -la /tmp/inspect | head -10
sudo umount /tmp/inspect
} || echo "(mount failed)"
sudo losetup -d "$loop"
echo ""
done
scripts/list-longhorn-backups.sh
Executable
+37
View File
@@ -0,0 +1,37 @@
#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="${BACKUP_DIR:-/mnt2/homey-backup}"
MANIFEST="$BACKUP_DIR/manifest.json"
echo "========================================"
echo " Longhorn Volume Backup List"
echo "========================================"
echo ""
if [[ ! -f "$MANIFEST" ]]; then
echo "No manifest found at $MANIFEST"
echo "Run backup-longhorn-to-disk.sh first."
exit 1
fi
echo "Backup timestamp: $(grep -oP '"backup_timestamp":\s*"\K[^"]+' "$MANIFEST")"
echo "Source: $(grep -oP '"source_path":\s*"\K[^"]+' "$MANIFEST")"
echo "Total volumes: $(grep -oP '"total_volumes":\s*\K[0-9]+' "$MANIFEST")"
echo "Total size: $(grep -oP '"total_size_bytes":\s*\K[0-9]+' "$MANIFEST" | numfmt --to=iec-i --suffix=B)"
echo ""
echo "Volumes:"
echo "----------------------------------------"
grep -A5 '"volumes"' "$MANIFEST" | grep -E '"friendly_name"|"size_human"' | \
while read -r name_line; read -r size_line; do
name=$(echo "$name_line" | grep -oP '"friendly_name":\s*"\K[^"]+')
size=$(echo "$size_line" | grep -oP '"size_human":\s*"\K[^"]+')
pvc=$(grep -B1 "$name_line" "$MANIFEST" | grep -oP '"pvc_id":\s*"\K[^"]+' || echo "")
printf " %-30s %10s %s\n" "$name" "$size" "$pvc"
done
echo ""
echo "Commands:"
echo " Mount: ./scripts/mount-longhorn-volume.sh <name>"
echo " Restore: ./scripts/restore-longhorn-volume.sh <name>"
scripts/longhorn-fuse.py
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env python3
import json
import os
import sys
from fuse import FUSE, FuseOSError, Operations
class LonghornBackupFS(Operations):
def __init__(self, backup_dir):
self.backup_dir = backup_dir
self.blocks_dir = f"{backup_dir}/blocks"
backup_cfg = f"{backup_dir}/backups/backup_backup-eac0221d1cab4a9c.cfg"
with open(backup_cfg) as f:
data = json.load(f)
self.size = int(data['Size'])
self.block_map = {b['Offset']: b['BlockChecksum'] for b in data['Blocks']}
self.block_size = 2097152 # 2MB
print(f"Volume size: {self.size}")
print(f"Blocks: {len(self.block_map)}")
def getattr(self, path, fh=None):
return {'st_size': self.size, 'st_mode': 0o100644, 'st_nlink': 1}
def read(self, path, size, offset, fh):
result = bytearray()
remaining = size
current_offset = offset
while remaining > 0:
block_start = (current_offset // self.block_size) * self.block_size
block_offset = current_offset - block_start
read_size = min(remaining, self.block_size - block_offset)
if block_start in self.block_map:
checksum = self.block_map[block_start]
block_path = f"{self.blocks_dir}/{checksum[:2]}/{checksum[2:4]}/{checksum}.blk"
if os.path.exists(block_path):
with open(block_path, 'rb') as f:
f.seek(block_offset)
result.extend(f.read(read_size))
else:
result.extend(b'\x00' * read_size)
else:
result.extend(b'\x00' * read_size)
current_offset += read_size
remaining -= read_size
return bytes(result)
if __name__ == '__main__':
if len(sys.argv) < 3:
print(f"Usage: {sys.argv[0]} <backup_dir> <mount_point>")
print(f"Example: {sys.argv[0]} /mnt2/backed-up-drive/backupstore/volumes/2c/df/pvc-5c1f48e3-346f-4c35-8e6a-8fc0c4c3a842 /tmp/longhorn-fuse")
sys.exit(1)
backup_dir = sys.argv[1]
mount_point = sys.argv[2]
os.makedirs(mount_point, exist_ok=True)
print(f"Mounting {backup_dir} at {mount_point}")
print("This creates a virtual block device file at the mount point")
print("Then run: sudo losetup -fP {mount_point}/volume.img && sudo mount /dev/loopX /mnt/point")
fs = LonghornBackupFS(backup_dir)
fuse = FUSE(fs, mount_point, nothreads=True, foreground=True, allow_other=True)
scripts/longhorn-nbdkit.py
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env python3
import nbdkit
import json
import os
backup_dir = "/mnt2/backed-up-drive/backupstore/volumes/2c/df/pvc-5c1f48e3-346f-4c35-8e6a-8fc0c4c3a842"
blocks_dir = f"{backup_dir}/blocks"
backup_cfg = f"{backup_dir}/backups/backup_backup-eac0221d1cab4a9c.cfg"
with open(backup_cfg) as f:
data = json.load(f)
size = int(data['Size'])
block_map = {b['Offset']: b['BlockChecksum'] for b in data['Blocks']}
block_size = 2097152
def thread_model():
return nbdkit.THREAD_MODEL_SERIALIZE_ALL_REQUESTS
def get_size():
return size
def pread(h, count, offset, flags):
result = bytearray()
remaining = count
current_offset = offset
while remaining > 0:
block_start = (current_offset // block_size) * block_size
block_offset = current_offset - block_start
read_size = min(remaining, block_size - block_offset)
if block_start in block_map:
checksum = block_map[block_start]
block_path = f"{blocks_dir}/{checksum[:2]}/{checksum[2:4]}/{checksum}.blk"
if os.path.exists(block_path):
with open(block_path, 'rb') as f:
f.seek(block_offset)
result.extend(f.read(read_size))
else:
result.extend(b'\x00' * read_size)
else:
result.extend(b'\x00' * read_size)
current_offset += read_size
remaining -= read_size
return bytes(result)
scripts/mount-longhorn-volume.sh
Executable
+146
View File
@@ -0,0 +1,146 @@
#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="${BACKUP_DIR:-/mnt2/homey-backup}"
MOUNT_BASE="${MOUNT_BASE:-/mnt/longhorn-volumes}"
usage() {
echo "Usage: $0 <pvc-name-or-friendly-name> [mount-point]"
echo ""
echo "Mounts a Longhorn volume backup for exploration."
echo ""
echo "Arguments:"
echo " pvc-name-or-friendly-name The PVC ID or friendly name (e.g., 'nextcloud-data-pvc')"
echo " mount-point Optional custom mount point (default: $MOUNT_BASE/<name>)"
echo ""
echo "Examples:"
echo " $0 nextcloud-data-pvc"
echo " $0 pvc-5c1f48e3-346f-4c35-8e6a-8fc0c4c3a842-96d72815"
echo " $0 nextcloud-data-pvc /mnt/my-mount"
echo ""
echo "To unmount, run:"
echo " sudo umount <mount-point>"
echo " sudo losetup -d /dev/loopX"
exit 1
}
if [[ $# -lt 1 ]]; then
usage
fi
SEARCH_NAME="$1"
CUSTOM_MOUNT="${2:-}"
MANIFEST="$BACKUP_DIR/manifest.json"
if [[ ! -f "$MANIFEST" ]]; then
echo "Error: Manifest not found at $MANIFEST"
echo "Make sure you've run the backup script first."
exit 1
fi
find_volume() {
local search="$1"
local found=""
while IFS= read -r line; do
pvc_id=$(echo "$line" | grep -oP '"pvc_id":\s*"\K[^"]+')
friendly=$(echo "$line" | grep -oP '"friendly_name":\s*"\K[^"]+')
if [[ "$pvc_id" == "$search" ]] || [[ "$friendly" == "$search" ]]; then
echo "$pvc_id:$friendly"
return 0
fi
done < <(grep -A6 '"volumes"' "$MANIFEST" | grep -E '"pvc_id"|"friendly_name"')
return 1
}
VOLUME_INFO=$(find_volume "$SEARCH_NAME")
if [[ -z "$VOLUME_INFO" ]]; then
echo "Error: Volume '$SEARCH_NAME' not found in manifest."
echo ""
echo "Available volumes:"
grep -oP '"friendly_name":\s*"\K[^"]+' "$MANIFEST" | while read -r name; do
echo " - $name"
done
exit 1
fi
PVC_ID="${VOLUME_INFO%%:*}"
FRIENDLY_NAME="${VOLUME_INFO#*:}"
VOLUME_DIR="$BACKUP_DIR/volumes/$PVC_ID"
if [[ ! -d "$VOLUME_DIR" ]]; then
echo "Error: Volume directory not found: $VOLUME_DIR"
exit 1
fi
VOLUME_HEAD=$(find "$VOLUME_DIR" -name "volume-head-*.img" | head -1)
if [[ -z "$VOLUME_HEAD" ]]; then
echo "Error: No volume-head-*.img file found in $VOLUME_DIR"
echo "Contents:"
ls -la "$VOLUME_DIR"
exit 1
fi
if [[ -n "$CUSTOM_MOUNT" ]]; then
MOUNT_POINT="$CUSTOM_MOUNT"
else
MOUNT_POINT="$MOUNT_BASE/$FRIENDLY_NAME"
fi
echo "========================================"
echo " Mount Longhorn Volume"
echo "========================================"
echo ""
echo "PVC ID: $PVC_ID"
echo "Name: $FRIENDLY_NAME"
echo "Volume file: $(basename "$VOLUME_HEAD")"
echo "Mount point: $MOUNT_POINT"
echo ""
LOOP_DEV=$(sudo losetup -fP --show "$VOLUME_HEAD")
echo "Attached to: $LOOP_DEV"
sudo mkdir -p "$MOUNT_POINT"
echo ""
echo "Mounting..."
if sudo mount "$LOOP_DEV" "$MOUNT_POINT" 2>/dev/null; then
echo ""
echo "========================================"
echo " Mounted Successfully!"
echo "========================================"
echo ""
echo "Mount point: $MOUNT_POINT"
echo "Loop device: $LOOP_DEV"
echo ""
echo "Contents:"
ls -la "$MOUNT_POINT" 2>/dev/null | head -20
echo ""
echo "To unmount:"
echo " sudo umount $MOUNT_POINT"
echo " sudo losetup -d $LOOP_DEV"
else
echo "Mount failed. Trying with filesystem detection..."
FS_TYPE=$(sudo blkid -o value -s TYPE "$LOOP_DEV" 2>/dev/null || echo "")
if [[ -n "$FS_TYPE" ]]; then
echo "Detected filesystem: $FS_TYPE"
sudo mount -t "$FS_TYPE" "$LOOP_DEV" "$MOUNT_POINT"
echo ""
echo "Mounted successfully at $MOUNT_POINT"
else
echo "Could not detect filesystem. Volume may be empty or corrupted."
echo ""
echo "Loop device: $LOOP_DEV"
echo "Run 'sudo blkid $LOOP_DEV' to inspect."
echo ""
echo "To detach:"
echo " sudo losetup -d $LOOP_DEV"
fi
fi
scripts/restore-fast.py
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env python3
import json
import os
import sys
import gzip
from concurrent.futures import ThreadPoolExecutor, as_completed
backup_dir = "/mnt2/backed-up-drive/backupstore/volumes/2c/df/pvc-5c1f48e3-346f-4c35-8e6a-8fc0c4c3a842"
output_img = "/mnt/nextcloud-restored.img"
backup_cfg = f"{backup_dir}/backups/backup_backup-eac0221d1cab4a9c.cfg"
blocks_dir = f"{backup_dir}/blocks"
with open(backup_cfg) as f:
data = json.load(f)
blocks = data['Blocks']
total = len(blocks)
size = int(data['Size'])
print(f"Volume size: {size // 1024 // 1024 // 1024} GB")
print(f"Block count: {total}")
os.makedirs(os.path.dirname(output_img) if os.path.dirname(output_img) else '.', exist_ok=True)
if not os.path.exists(output_img):
import subprocess
subprocess.run(['truncate', '-s', str(size), output_img], check=True)
with open(output_img, 'r+b') as img:
for i, block in enumerate(blocks):
offset = block['Offset']
checksum = block['BlockChecksum']
block_path = f"{blocks_dir}/{checksum[:2]}/{checksum[2:4]}/{checksum}.blk"
if os.path.exists(block_path):
with gzip.open(block_path, 'rb') as bf:
img.seek(offset)
img.write(bf.read())
if (i + 1) % 500 == 0:
percent = (i + 1) * 100 // total
bar = '=' * (percent // 2) + ' ' * (50 - percent // 2)
sys.stdout.write(f"\r[{bar}] {percent}% ({i + 1}/{total})")
sys.stdout.flush()
print(f"\nDone! Image: {output_img}")
scripts/restore-longhorn-backup.sh
Executable
+94
View File
@@ -0,0 +1,94 @@
#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="${1:-/mnt2/backed-up-drive/backupstore/volumes/2c/df/pvc-5c1f48e3-346f-4c35-8e6a-8fc0c4c3a842}"
OUTPUT_IMG="${2:-./nextcloud-data-restored.img}"
BACKUP_CFG="$BACKUP_DIR/backups/backup_backup-eac0221d1cab4a9c.cfg"
BLOCKS_DIR="$BACKUP_DIR/blocks"
if [[ ! -f "$BACKUP_CFG" ]]; then
echo "Error: Backup config not found at $BACKUP_CFG"
exit 1
fi
echo "========================================"
echo " Longhorn Backup Restore Tool"
echo "========================================"
echo ""
echo "Backup: $BACKUP_DIR"
echo "Output: $OUTPUT_IMG"
echo ""
SIZE=$(python3 -c "import json; print(json.load(open('$BACKUP_CFG'))['Size'])")
BLOCK_COUNT=$(python3 -c "import json; print(len(json.load(open('$BACKUP_CFG'))['Blocks']))")
echo "Volume size: $((SIZE / 1024 / 1024 / 1024)) GB ($SIZE bytes)"
echo "Block count: $BLOCK_COUNT"
echo ""
read -p "Continue? [y/N] " -n 1 -r
echo ""
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Aborted."
exit 1
fi
echo ""
echo "Creating sparse image..."
truncate -s "$SIZE" "$OUTPUT_IMG"
echo "Restoring blocks..."
python3 << 'PYEOF'
import json
import os
import sys
backup_cfg = os.environ['BACKUP_CFG']
blocks_dir = os.environ['BLOCKS_DIR']
output_img = os.environ['OUTPUT_IMG']
with open(backup_cfg) as f:
data = json.load(f)
blocks = data['Blocks']
total = len(blocks)
with open(output_img, 'r+b') as img:
for i, block in enumerate(blocks):
offset = block['Offset']
checksum = block['BlockChecksum']
block_path = f"{blocks_dir}/{checksum[:2]}/{checksum[2:4]}/{checksum}.blk"
if not os.path.exists(block_path):
print(f"Warning: Block not found: {checksum}")
continue
with open(block_path, 'rb') as bf:
img.seek(offset)
img.write(bf.read())
if (i + 1) % 1000 == 0 or i + 1 == total:
percent = (i + 1) * 100 // total
bar = '=' * (percent // 2) + ' ' * (50 - percent // 2)
sys.stdout.write(f"\r[{bar}] {percent}% ({i + 1}/{total})")
sys.stdout.flush()
print()
PYEOF
echo ""
echo "========================================"
echo " Restore Complete!"
echo "========================================"
echo ""
echo "Image: $OUTPUT_IMG"
echo "Size: $(du -sh "$OUTPUT_IMG" | cut -f1)"
echo ""
echo "To mount:"
echo " sudo losetup -fP $OUTPUT_IMG"
echo " sudo mount /dev/loopX /mnt/point"
echo ""
echo "Or directly:"
echo " sudo mount -o loop $OUTPUT_IMG /mnt/point"
scripts/restore-longhorn-volume.sh
Executable
+135
View File
@@ -0,0 +1,135 @@
#!/usr/bin/env bash
set -euo pipefail
BACKUP_DIR="${BACKUP_DIR:-/mnt2/homey-backup}"
RESTORE_BASE="${RESTORE_BASE:-/mnt/replicas}"
usage() {
echo "Usage: $0 <pvc-name-or-friendly-name> [--dry-run]"
echo ""
echo "Restores a Longhorn volume backup to the replicas directory."
echo ""
echo "Arguments:"
echo " pvc-name-or-friendly-name The PVC ID or friendly name"
echo " --dry-run Show what would be done without copying"
echo ""
echo "Examples:"
echo " $0 nextcloud-data-pvc"
echo " $0 nextcloud-data-pvc --dry-run"
echo ""
echo "WARNING: This will overwrite existing data in $RESTORE_BASE"
exit 1
}
if [[ $# -lt 1 ]]; then
usage
fi
SEARCH_NAME="$1"
DRY_RUN=false
if [[ "${2:-}" == "--dry-run" ]]; then
DRY_RUN=true
fi
MANIFEST="$BACKUP_DIR/manifest.json"
if [[ ! -f "$MANIFEST" ]]; then
echo "Error: Manifest not found at $MANIFEST_DIR"
exit 1
fi
find_volume() {
local search="$1"
while IFS= read -r line; do
pvc_id=$(echo "$line" | grep -oP '"pvc_id":\s*"\K[^"]+')
friendly=$(echo "$line" | grep -oP '"friendly_name":\s*"\K[^"]+')
if [[ "$pvc_id" == "$search" ]] || [[ "$friendly" == "$search" ]]; then
echo "$pvc_id:$friendly"
return 0
fi
done < <(grep -A6 '"volumes"' "$MANIFEST" | grep -E '"pvc_id"|"friendly_name"')
return 1
}
VOLUME_INFO=$(find_volume "$SEARCH_NAME")
if [[ -z "$VOLUME_INFO" ]]; then
echo "Error: Volume '$SEARCH_NAME' not found in manifest."
echo ""
echo "Available volumes:"
grep -oP '"friendly_name":\s*"\K[^"]+' "$MANIFEST" | while read -r name; do
echo " - $name"
done
exit 1
fi
PVC_ID="${VOLUME_INFO%%:*}"
FRIENDLY_NAME="${VOLUME_INFO#*:}"
BACKUP_VOLUME_DIR="$BACKUP_DIR/volumes/$PVC_ID"
RESTORE_VOLUME_DIR="$RESTORE_BASE/$PVC_ID"
echo "========================================"
echo " Restore Longhorn Volume"
echo "========================================"
echo ""
echo "PVC ID: $PVC_ID"
echo "Name: $FRIENDLY_NAME"
echo "Source: $BACKUP_VOLUME_DIR"
echo "Destination: $RESTORE_VOLUME_DIR"
echo "Dry run: $DRY_RUN"
echo ""
if [[ "$DRY_RUN" == "true" ]]; then
echo "[DRY RUN] Would copy:"
du -sh "$BACKUP_VOLUME_DIR" 2>/dev/null || echo " (size unknown)"
echo ""
echo "Files to copy:"
find "$BACKUP_VOLUME_DIR" -type f | head -20
exit 0
fi
if [[ -d "$RESTORE_VOLUME_DIR" ]]; then
echo "WARNING: Destination already exists!"
echo ""
read -p "Overwrite existing data? [y/N] " -n 1 -r
echo ""
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Aborted."
exit 1
fi
echo ""
echo "Removing existing data..."
sudo rm -rf "$RESTORE_VOLUME_DIR"
fi
echo "Creating destination directory..."
sudo mkdir -p "$RESTORE_VOLUME_DIR"
echo "Copying volume data..."
sudo rsync -a --no-owner --no-group --info=progress2 \
"$BACKUP_VOLUME_DIR/" \
"$RESTORE_VOLUME_DIR/"
echo ""
echo "Setting permissions..."
sudo chmod 700 "$RESTORE_VOLUME_DIR"
echo ""
echo "========================================"
echo " Restore Complete!"
echo "========================================"
echo ""
echo "Restored to: $RESTORE_VOLUME_DIR"
echo ""
echo "Size:"
sudo du -sh "$RESTORE_VOLUME_DIR"
echo ""
echo "Next steps:"
echo "1. Ensure Longhorn is configured to use $RESTORE_BASE"
echo "2. Restart Longhorn or the affected pod"
echo "3. Verify data integrity"
templates/auth.yaml
+118
View File
@@ -548,3 +548,121 @@ spec:
port: port:
number: 80 number: 80
--- ---
#START RADICALE
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: radicale-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: longhorn
---
apiVersion: v1
kind: ConfigMap
metadata:
name: radicale-conf
labels:
app: radicale
data:
config: |-
{{ tpl (.Files.Get "files/radicale-configmap.ini" | indent 4) . }}
---
{{- $_ := set $ "homey_radicale_basic_auth" (include "homey.lookuporgensecret" (merge (dict "secretname" "radicale-basic-auth") $))}}
{{ include "homey.randomsecret" (merge (dict "secretname" "radicale-basic-auth" "secretval" .homey_radicale_basic_auth) $) }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: radicale
labels:
app.kubernetes.io/name: radicale
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: radicale
template:
metadata:
labels:
app.kubernetes.io/name: radicale
spec:
containers:
- name: radicale
image: tomsquest/docker-radicale
imagePullPolicy: IfNotPresent
ports:
- name: dav
containerPort: 5232
protocol: TCP
volumeMounts:
- name: collections
mountPath: /data/collections
- name: config
mountPath: /config/config
subPath: config
readOnly: true
restartPolicy: Always
volumes:
- name: collections
persistentVolumeClaim:
claimName: radicale-pvc
- name: config
configMap:
name: radicale-conf
---
apiVersion: v1
kind: Service
metadata:
name: radicale
labels:
app.kubernetes.io/name: radicale
spec:
type: ClusterIP
ports:
- name: dav
port: 5232
targetPort: dav
selector:
app.kubernetes.io/name: radicale
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: radicale-dav
annotations:
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/auth-method: GET
nginx.ingress.kubernetes.io/auth-url: http://authelia.{{ .Release.Namespace }}.svc.cluster.local:9091/api/verify
nginx.ingress.kubernetes.io/auth-signin: https://auth.{{ .Values.homey.url }}?rm=$request_method
nginx.ingress.kubernetes.io/auth-response-headers: Remote-User,Remote-Name,Remote-Groups,Remote-Email
nginx.ingress.kubernetes.io/auth-snippet: |
proxy_set_header X-Forwarded-Method $request_method;
auth_request_set $user $upstream_http_remote_user;
auth_request_set $groups $upstream_http_remote_groups;
auth_request_set $name $upstream_http_remote_name;
auth_request_set $email $upstream_http_remote_email;
proxy_set_header X-Remote-User $user;
proxy_set_header X-Remote-Fullname $name;
proxy_set_header X-Remote-Email $email;
spec:
ingressClassName: {{ .Values.homey.ingress_class }}
tls:
- hosts:
- dav.{{ .Values.homey.url }}
secretName: {{ .Values.homey.certname }}
rules:
- host: dav.{{ .Values.homey.url }}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: radicale
port:
number: 5232
templates/media.yaml
+12 -5
View File
@@ -51,8 +51,18 @@ spec:
spec: spec:
containers: containers:
- name: jellyfin - name: jellyfin
image: docker.io/jellyfin/jellyfin image: jellyfin/jellyfin:10.11.6
imagePullPolicy: "IfNotPresent" ports:
- containerPort: 8096
name: http
env:
- name: PGUID
value: "1000"
- name: PUID
value: "1000"
- name: JELLYFIN_PublishedServerUrl
value: jellyfin.{{ .Values.homey.url }}
imagePullPolicy: "Always"
volumeMounts: volumeMounts:
- name: jellyfin-volume-config - name: jellyfin-volume-config
mountPath: "/config" mountPath: "/config"
@@ -63,9 +73,6 @@ spec:
- name: jellyfin-volume-data - name: jellyfin-volume-data
mountPath: "/data/tvshows" mountPath: "/data/tvshows"
subPath: downloads/tvshows subPath: downloads/tvshows
env:
- name: JELLYFIN_PublishedServerUrl
value: jellyfin.{{ .Values.homey.url }}
volumes: volumes:
- name: jellyfin-volume-config - name: jellyfin-volume-config
persistentVolumeClaim: persistentVolumeClaim:
values.yaml
+1 -1
View File
@@ -59,7 +59,7 @@ affinity: {}
homey: homey:
organization: "Zakobar Home Server" organization: "Zakobar Home Server"
url: zakobar.com url: zakobar.com
ip: 10.0.0.100 ip: 192.168.1.100
certname: zakobarcert certname: zakobarcert
ingress_class: nginx ingress_class: nginx