Elasticsearch Snapshot to File System and Upload to Cloud
A snapshot is a fill-in of a running Elasticsearch cluster. You can use snapshots to:
- Regularly support a cluster with no reanimation
- Recover information later deletion or a hardware failure
- Transfer data betwixt clusters
- Reduce your storage costs by using searchable snapshots in the cold and frozen data tiers
The snapshot workflowedit
Elasticsearch stores snapshots in an off-cluster storage location called a snapshot repository. Earlier you can take or restore snapshots, you must annals a snapshot repository on the cluster. Elasticsearch supports several repository types with cloud storage options, including:
- AWS S3
- Google Cloud Storage (GCS)
- Microsoft Azure
After you annals a snapshot repository, you tin use snapshot lifecycle management (SLM) to automatically take and manage snapshots. You can then restore a snapshot to recover or transfer its data.
Snapshot contentsedit
By default, a snapshot of a cluster contains the cluster state, all regular data streams, and all regular indices. The cluster state includes:
- Persistent cluster settings
- Index templates
- Legacy index templates
- Ingest pipelines
- ILM policies
- For snapshots taken after 7.12.0, feature states
You lot can also take snapshots of only specific information streams or indices in the cluster. A snapshot that includes a data stream or alphabetize automatically includes its aliases. When y'all restore a snapshot, y'all can choose whether to restore these aliases.
Snapshots don't contain or back up:
- Transient cluster settings
- Registered snapshot repositories
- Node configuration files
- Security configuration files
Characteristic statesedit
A characteristic state contains the indices and data streams used to shop configurations, history, and other data for an Elastic feature, such every bit Elasticsearch security or Kibana.
A feature state typically includes one or more system indices or system data streams. It may also include regular indices and information streams used by the feature. For example, a characteristic state may include a regular index that contains the feature'south execution history. Storing this history in a regular index lets yous more hands search it.
In Elasticsearch 8.0 and later versions, feature states are the only way to support and restore system indices and system data streams.
How snapshots workedit
Snapshots are automatically deduplicated to save storage infinite and reduce network transfer costs. To back up an alphabetize, a snapshot makes a copy of the index'south segments and stores them in the snapshot repository. Since segments are immutable, the snapshot only needs to re-create any new segments created since the repository'south final snapshot.
Each snapshot is too logically independent. When you delete a snapshot, Elasticsearch only deletes the segments used exclusively by that snapshot. Elasticsearch doesn't delete segments used by other snapshots in the repository.
Snapshots and shard allocationedit
A snapshot copies segments from an index's primary shards. When you start a snapshot, Elasticsearch immediately starts copying the segments of whatever bachelor principal shards. If a shard is starting or relocating, Elasticsearch will await for these processes to complete earlier copying the shard'southward segments. If ane or more than primary shards aren't bachelor, the snapshot effort fails.
Once a snapshot begins copying a shard's segments, Elasticsearch won't motion the shard to another node, fifty-fifty if rebalancing or shard allocation settings would typically trigger reallocation. Elasticsearch volition only motion the shard later the snapshot finishes copying the shard'due south data.
Snapshot start and end timesedit
A snapshot doesn't represent a cluster at a precise signal in fourth dimension. Instead, each snapshot includes a get-go and stop time. The snapshot represents a view of each shard's data at some point between these two times.
Snapshot compatibilityedit
To restore a snapshot to a cluster, the versions for the snapshot, cluster, and any restored indices must be compatible.
Snapshot version compatibilityedit
| Cluster version | ||||
| Snapshot version | vi.8 | vii.0–7.i | 7.2–7.17 | 8.0–8.1 |
| five.0–5.6 | | | | |
| 6.0–6.7 | | | | |
| 6.viii | | | | |
| 7.0–7.i | | | | |
| seven.2–seven.17 | | | | |
| 8.0–8.i | | | | |
Y'all can't restore a snapshot to an before version of Elasticsearch. For example, you lot can't restore a snapshot taken in 7.six.0 to a cluster running 7.5.0.
Alphabetize compatibilityedit
Any index you restore from a snapshot must also exist compatible with the current cluster's version. If you try to restore an index created in an incompatible version, the restore attempt volition fail.
| Cluster version | ||||
| Alphabetize creation version | vi.8 | seven.0–7.1 | 7.2–7.17 | 8.0–eight.i |
| 5.0–five.6 | | | | |
| 6.0–six.7 | | | | |
| half-dozen.viii | | | | |
| 7.0–7.one | | | | |
| 7.2–7.17 | | | | |
| 8.0–8.1 | | | | |
You tin can't restore an alphabetize to an earlier version of Elasticsearch. For example, yous can't restore an alphabetize created in 7.6.0 to a cluster running seven.five.0.
A compatible snapshot can contain indices created in an incompatible version. For example, a snapshot of a seven.17 cluster can incorporate an alphabetize created in 6.8. If you endeavour to restore the 6.8 index to an viii.i cluster, the restore attempt will neglect. Keep this in mind if you take a snapshot earlier upgrading a cluster.
Equally a workaround, you tin can first restore the alphabetize to another cluster running the latest version of Elasticsearch that's compatible with both the alphabetize and your current cluster. You tin and so use reindex-from-remote to rebuild the index on your current cluster. Reindex from remote is only possible if the index'southward _source is enabled.
Reindexing from remote can take significantly longer than restoring a snapshot. Earlier you lot commencement, test the reindex from remote process with a subset of the data to approximate your time requirements.
Warningsedit
Other backup methodsedit
Taking a snapshot is the only reliable and supported fashion to back up a cluster. Yous cannot support an Elasticsearch cluster past making copies of the data directories of its nodes. There are no supported methods to restore whatsoever data from a filesystem-level backup. If you try to restore a cluster from such a backup, information technology may neglect with reports of corruption or missing files or other data inconsistencies, or it may announced to have succeeded having silently lost some of your data.
A copy of the data directories of a cluster's nodes does not work as a fill-in because it is non a consistent representation of their contents at a single point in time. You cannot fix this by shutting downwards nodes while making the copies, nor by taking diminutive filesystem-level snapshots, considering Elasticsearch has consistency requirements that span the whole cluster. You must use the congenital-in snapshot functionality for cluster backups.
Repository contentsedit
Don't alter anything within the repository or run processes that might interfere with its contents. If something other than Elasticsearch modifies the contents of the repository and so future snapshot or restore operations may fail, reporting corruption or other data inconsistencies, or may appear to succeed having silently lost some of your data.
Y'all may even so safely restore a repository from a backup as long as
- The repository is not registered with Elasticsearch while you are restoring its contents.
- When yous accept finished restoring the repository its contents are exactly as they were when yous took the backup.
Additionally, snapshots may incorporate security-sensitive data, which you may wish to store in a dedicated repository.
chomleyalearright.blogspot.com
Source: https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html