PC Atomic Sync 3.8 serial key or number

3.4. Connect mutations

3.5. Shared resolvers

1. Introduction

This document enumerates the requirements that must be met in order for devices to be compatible with Android 11.

The use of “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” is per the IETF standard defined in RFC2119.

As used in this document, a “device implementer” or “implementer” is a person or organization developing a hardware/software solution running Android 11. A “device implementation” or “implementation" is the hardware/software solution so developed.

To be considered compatible with Android 11, device implementations MUST meet the requirements presented in this Compatibility Definition, including any documents incorporated via reference.

Where this definition or the software tests described in section 10 is silent, ambiguous, or incomplete, it is the responsibility of the device implementer to ensure compatibility with existing implementations.

For this reason, the Android Open Source Project is both the reference and preferred implementation of Android. Device implementers are STRONGLY RECOMMENDED to base their implementations to the greatest extent possible on the “upstream” source code available from the Android Open Source Project. While some components can hypothetically be replaced with alternate implementations, it is STRONGLY RECOMMENDED to not follow this practice, as passing the software tests will become substantially more difficult. It is the implementer’s responsibility to ensure full behavioral compatibility with the standard Android implementation, including and beyond the Compatibility Test Suite. Finally, note that certain component substitutions and modifications are explicitly forbidden by this document.

Many of the resources linked to in this document are derived directly or indirectly from the Android SDK and will be functionally identical to the information in that SDK’s documentation. In any cases where this Compatibility Definition or the Compatibility Test Suite disagrees with the SDK documentation, the SDK documentation is considered authoritative. Any technical details provided in the linked resources throughout this document are considered by inclusion to be part of this Compatibility Definition.

1.1 Document Structure

1.1.1. Requirements by Device Type

Section 2 contains all of the requirements that apply to a specific device type. Each subsection of Section 2 is dedicated to a specific device type.

All the other requirements, that universally apply to any Android device implementations, are listed in the sections after Section 2. These requirements are referenced as "Core Requirements" in this document.

1.1.2. Requirement ID

Requirement ID is assigned for MUST requirements.

• The ID is assigned for MUST requirements only.
• STRONGLY RECOMMENDED requirements are marked as [SR] but ID is not assigned.
• The ID consists of : Device Type ID - Condition ID - Requirement ID (e.g. C-0-1).

Each ID is defined as below:

• Device Type ID (see more in 2. Device Types)
  • C: Core (Requirements that are applied to any Android device implementations)
  • H: Android Handheld device
  • T: Android Television device
  • A: Android Automotive implementation
  • W: Android Watch implementation
  • Tab: Android Tablet implementation
• Condition ID
  • When the requirement is unconditional, this ID is set as 0.
  • When the requirement is conditional, 1 is assigned for the 1st condition and the number increments by 1 within the same section and the same device type.
• Requirement ID
  • This ID starts from 1 and increments by 1 within the same section and the same condition.

1.1.3. Requirement ID in Section 2

The Requirement ID in Section 2 starts with the corresponding section ID that is followed by the Requirement ID described above.

• The ID in Section 2 consists of : Section ID / Device Type ID - Condition ID - Requirement ID (e.g. 7.4.3/A-0-1).

2. Device Types

While the Android Open Source Project provides a software stack that can be used for a variety of device types and form factors, there are a few device types that have a relatively better established application distribution ecosystem.

This section describes those device types, and additional requirements and recommendations applicable for each device type.

All Android device implementations that do not fit into any of the described device types MUST still meet all requirements in the other sections of this Compatibility Definition.

2.1 Device Configurations

For the major differences in hardware configuration by device type, see the device-specific requirements that follow in this section.

2.2. Handheld Requirements

An Android Handheld device refers to an Android device implementation that is typically used by holding it in the hand, such as an mp3 player, phone, or tablet.

Android device implementations are classified as a Handheld if they meet all the following criteria:

• Have a power source that provides mobility, such as a battery.
• Have a physical diagonal screen size in the range of 3.3 inches (or 2.5 inches for devices which launched on an API level earlier than Android 11) to 8 inches.

The additional requirements in the rest of this section are specific to Android Handheld device implementations.

2.2.1. Hardware

Handheld device implementations:

• [7.1.1.1/H-0-1] MUST have at least one Android-compatible display that meets all requirements described on this document.
• [7.1.1.3/H-SR] Are STRONGLY RECOMMENDED to provide users an affordance to change the display size (screen density).

If Handheld device implementations support software screen rotation, they:

• [7.1.1.1/H-1-1]* MUST make the logical screen that is made available for third party applications be at least 2 inches on the short edge(s) and 2.7 inches on the long edge(s). Devices which launched on an API level earlier than that of this document are exempted from this requirement.

If Handheld device implementations do not support software screen rotation, they:

• [7.1.1.1/H-2-1]* MUST make the logical screen that is made available for third party applications be at least 2.7 inches on the short edge(s). Devices which launched on an API level earlier than that of this document are exempted from this requirement.

If Handheld device implementations claim support for high dynamic range displays through , they:

• [7.1.4.5/H-1-1] MUST advertise support for the , , , , and extensions.

Handheld device implementations:

• [7.1.4.6/H-0-1] MUST report whether the device supports the GPU profiling capability via a system property .

If Handheld device implementations declare support via a system property , they:

Handheld device implementations:

• [7.1.5/H-0-1] MUST include support for legacy application compatibility mode as implemented by the upstream Android open source code. That is, device implementations MUST NOT alter the triggers or thresholds at which compatibility mode is activated, and MUST NOT alter the behavior of the compatibility mode itself.
• [7.2.1/H-0-1] MUST include support for third-party Input Method Editor (IME) applications.
• [7.2.3/H-0-3] MUST provide the Home function on all the Android-compatible displays that provide the home screen.
• [7.2.3/H-0-4] MUST provide the Back function on all the Android-compatible displays and the Recents function on at least one of the Android-compatible displays.
• [7.2.3/H-0-2] MUST send both the normal and long press event of the Back function () to the foreground application. These events MUST NOT be consumed by the system and CAN be triggered by outside of the Android device (e.g. external hardware keyboard connected to the Android device).
• [7.2.4/H-0-1] MUST support touchscreen input.
• [7.2.4/H-SR] Are STRONGLY RECOMMENDED to launch the user-selected assist app, in other words the app that implements VoiceInteractionService, or an activity handling the on long-press of or if the foreground activity does not handle those long-press events.
• [7.3.1/H-SR] Are STRONGLY RECOMMENDED to include a 3-axis accelerometer.

If Handheld device implementations include a 3-axis accelerometer, they:

• [7.3.1/H-1-1] MUST be able to report events up to a frequency of at least 100 Hz.

If Handheld device implementations include a GPS/GNSS receiver and report the capability to applications through the feature flag, they:

• [7.3.3/H-2-1] MUST report GNSS measurements, as soon as they are found, even if a location calculated from GPS/GNSS is not yet reported.
• [7.3.3/H-2-2] MUST report GNSS pseudoranges and pseudorange rates, that, in open-sky conditions after determining the location, while stationary or moving with less than 0.2 meter per second squared of acceleration, are sufficient to calculate position within 20 meters, and speed within 0.2 meters per second, at least 95% of the time.

If Handheld device implementations include a 3-axis gyroscope, they:

• [7.3.4/H-3-1] MUST be able to report events up to a frequency of at least 100 Hz.
• [7.3.4/H-3-2] MUST be capable of measuring orientation changes up to 1000 degrees per second.

Handheld device implementations that can make a voice call and indicate any value other than in :

• [7.3.8/H] SHOULD include a proximity sensor.

Handheld device implementations:

• [7.3.11/H-SR] Are RECOMMENDED to support pose sensor with 6 degrees of freedom.
• [7.4.3/H] SHOULD include support for Bluetooth and Bluetooth LE.

If Handheld device implementations include a metered connection, they:

• [7.4.7/H-1-1] MUST provide the data saver mode.

If Handheld device implementations include a logical camera device that lists capabilities using , they:

• [7.5.4/H-1-1] MUST have normal field of view (FOV) by default and it MUST be between 50 and 90 degrees.

Handheld device implementations:

• [7.6.1/H-0-1] MUST have at least 4 GB of non-volatile storage available for application private data (a.k.a. "/data" partition).
• [7.6.1/H-0-2] MUST return “true” for when there is less than 1GB of memory available to the kernel and userspace.

If Handheld device implementations declare support of only a 32-bit ABI:

• [7.6.1/H-1-1] The memory available to the kernel and userspace MUST be at least 416MB if the default display uses framebuffer resolutions up to qHD (e.g. FWVGA).

• [7.6.1/H-2-1] The memory available to the kernel and userspace MUST be at least 592MB if the default display uses framebuffer resolutions up to HD+ (e.g. HD, WSVGA).

• [7.6.1/H-3-1] The memory available to the kernel and userspace MUST be at least 896MB if the default display uses framebuffer resolutions up to FHD (e.g. WSXGA+).

• [7.6.1/H-4-1] The memory available to the kernel and userspace MUST be at least 1344MB if the default display uses framebuffer resolutions up to QHD (e.g. QWXGA).

If Handheld device implementations declare support of 32-bit and 64-bit ABIs:

• [7.6.1/H-5-1] The memory available to the kernel and userspace MUST be at least 816MB if the default display uses framebuffer resolutions up to qHD (e.g. FWVGA).

• [7.6.1/H-6-1] The memory available to the kernel and userspace MUST be at least 944MB if the default display uses framebuffer resolutions up to HD+ (e.g. HD, WSVGA).

• [7.6.1/H-7-1] The memory available to the kernel and userspace MUST be at least 1280MB if the default display uses framebuffer resolutions up to FHD (e.g. WSXGA+).

• [7.6.1/H-8-1] The memory available to the kernel and userspace MUST be at least 1824MB if the default display uses framebuffer resolutions up to QHD (e.g. QWXGA).

Note that the "memory available to the kernel and userspace" above refers to the memory space provided in addition to any memory already dedicated to hardware components such as radio, video, and so on that are not under the kernel’s control on device implementations.

If Handheld device implementations include less than or equal to 1GB of memory available to the kernel and userspace, they:

• [7.6.1/H-9-1] MUST declare the feature flag .
• [7.6.1/H-9-2] MUST have at least 1.1 GB of non-volatile storage for application private data (a.k.a. "/data" partition).

If Handheld device implementations include more than 1GB of memory available to the kernel and userspace, they:

• [7.6.1/H-10-1] MUST have at least 4GB of non-volatile storage available for application private data (a.k.a. "/data" partition).
• SHOULD declare the feature flag .

Handheld device implementations:

• [7.6.2/H-0-1] MUST NOT provide an application shared storage smaller than 1 GiB.
• [7.7.1/H] SHOULD include a USB port supporting peripheral mode.

If handheld device implementations include a USB port supporting peripheral mode, they:

• [7.7.1/H-1-1] MUST implement the Android Open Accessory (AOA) API.

If Handheld device implementations include a USB port supporting host mode, they:

• [7.7.2/H-1-1] MUST implement the USB audio class as documented in the Android SDK documentation.

Handheld device implementations:

• [7.8.1/H-0-1] MUST include a microphone.
• [7.8.2/H-0-1] MUST have an audio output and declare .

If Handheld device implementations are capable of meeting all the performance requirements for supporting VR mode and include support for it, they:

• [7.9.1/H-1-1] MUST declare the feature flag.
• [7.9.1/H-1-2] MUST include an application implementing that can be enabled by VR applications via .

If Handheld device implementations include one or more USB-C port(s) in host mode and implement (USB audio class), in addition to requirements in section 7.7.2, they:

• [7.8.2.2/H-1-1] MUST provide the following software mapping of HID codes:

• [7.8.2.2/H-1-2] MUST trigger ACTION_HEADSET_PLUG upon a plug insert, but only after the USB audio interfaces and endpoints have been properly enumerated in order to identify the type of terminal connected.

When the USB audio terminal types 0x0302 is detected, they:

• [7.8.2.2/H-2-1] MUST broadcast Intent ACTION_HEADSET_PLUG with "microphone" extra set to 0.

When the USB audio terminal types 0x0402 is detected, they:

• [7.8.2.2/H-3-1] MUST broadcast Intent ACTION_HEADSET_PLUG with "microphone" extra set to 1.

When API AudioManager.getDevices() is called while the USB peripheral is connected they:

• [7.8.2.2/H-4-1] MUST list a device of type AudioDeviceInfo.TYPE_USB_HEADSET and role isSink() if the USB audio terminal type field is 0x0302.

• [7.8.2.2/H-4-2] MUST list a device of type AudioDeviceInfo.TYPE_USB_HEADSET and role isSink() if the USB audio terminal type field is 0x0402.

• [7.8.2.2/H-4-3] MUST list a device of type AudioDeviceInfo.TYPE_USB_HEADSET and role isSource() if the USB audio terminal type field is 0x0402.

• [7.8.2.2/H-4-4] MUST list a device of type AudioDeviceInfo.TYPE_USB_DEVICE and role isSink() if the USB audio terminal type field is 0x603.

• [7.8.2.2/H-4-5] MUST list a device of type AudioDeviceInfo.TYPE_USB_DEVICE and role isSource() if the USB audio terminal type field is 0x604.

• [7.8.2.2/H-4-6] MUST list a device of type AudioDeviceInfo.TYPE_USB_DEVICE and role isSink() if the USB audio terminal type field is 0x400.

• [7.8.2.2/H-4-7] MUST list a device of type AudioDeviceInfo.TYPE_USB_DEVICE and role isSource() if the USB audio terminal type field is 0x400.

• [7.8.2.2/H-SR] Are STRONGLY RECOMMENDED upon connection of a USB-C audio peripheral, to perform enumeration of USB descriptors, identify terminal types and broadcast Intent ACTION_HEADSET_PLUG in less than 1000 milliseconds.

If Handheld device implementations include at least one haptic actuator, they:

• [7.10/H-SR]* Are STRONGLY RECOMMENDED NOT to use an eccentric rotating mass (ERM) haptic actuator(vibrator).
• [7.10/H]* SHOULD position the placement of the actuator near the location where the device is typically held or touched by hands.
• [7.10/H-SR]* Are STRONGLY RECOMMENDED to implement all public constants for clear haptics in android.view.HapticFeedbackConstants namely (CLOCK_TICK, CONTEXT_CLICK, KEYBOARD_PRESS, KEYBOARD_RELEASE, KEYBOARD_TAP, LONG_PRESS, TEXT_HANDLE_MOVE, VIRTUAL_KEY, VIRTUAL_KEY_RELEASE, CONFIRM, REJECT, GESTURE_START and GESTURE_END).
• [7.10/H-SR]* Are STRONGLY RECOMMENDED to implement all public constants for clear haptics in android.os.VibrationEffect namely (EFFECT_TICK, EFFECT_CLICK, EFFECT_HEAVY_CLICK and EFFECT_DOUBLE_CLICK) and all public constants for rich haptics in android.os.VibrationEffect.Composition namely (PRIMITIVE_CLICK and PRIMITIVE_TICK).
• [7.10/H-SR]* Are STRONGLY RECOMMENDED to use these linked haptic constants mappings.
• [7.10/H-SR]* Are STRONGLY RECOMMENDED to follow quality assessment for createOneShot() and createWaveform() API's.
• [7.10/H-SR]* Are STRONGLY RECOMMENDED to verify the capabilities for amplitude scalability by running android.os.Vibrator.hasAmplitudeControl().

Linear resonant actuator (LRA) is a single mass spring system which has a dominant resonant frequency where the mass translates in the direction of desired motion.

If Handheld device implementations include at least one linear resonant actuator, they:

• [7.10/H]* SHOULD move the haptic actuator in the X-axis of portrait orientation.

If Handheld device implementations have a haptic actuator which is X-axis Linear resonant actuator (LRA), they:

• [7.10/H-SR]* Are STRONGLY RECOMMENDED to have the resonant frequency of the X-axis LRA be under 200 Hz.

If handheld device implementations follow haptic constants mapping, they:

2.2.2. Multimedia

Handheld device implementations MUST support the following audio encoding and decoding formats and make them available to third-party applications:

• [5.1/H-0-1] AMR-NB
• [5.1/H-0-2] AMR-WB
• [5.1/H-0-3] MPEG-4 AAC Profile (AAC LC)
• [5.1/H-0-4] MPEG-4 HE AAC Profile (AAC+)
• [5.1/H-0-5] AAC ELD (enhanced low delay AAC)

Handheld device implementations MUST support the following video encoding formats and make them available to third-party applications:

• [5.2/H-0-1] H.264 AVC
• [5.2/H-0-2] VP8

Handheld device implementations MUST support the following video decoding formats and make them available to third-party applications:

• [5.3/H-0-1] H.264 AVC
• [5.3/H-0-2] H.265 HEVC
• [5.3/H-0-3] MPEG-4 SP
• [5.3/H-0-4] VP8
• [5.3/H-0-5] VP9

2.2.3. Software

Handheld device implementations:

• [3.2.3.1/H-0-1] MUST have an application that handles the , , , and intents as described in the SDK documents, and provide the user affordance to access the document provider data by using API.
• [3.2.3.1/H-0-2]* MUST preload one or more applications or service components with an intent handler, for all the public intent filter patterns defined by the following application intents listed here.
• [3.2.3.1/H-SR] Are STRONGLY RECOMMENDED to preload an email application which can handle ACTION_SENDTO or ACTION_SEND or ACTION_SEND_MULTIPLE intents to send an email.
• [3.4.1/H-0-1] MUST provide a complete implementation of the API.
• [3.4.2/H-0-1] MUST include a standalone Browser application for general user web browsing.
• [3.8.1/H-SR] Are STRONGLY RECOMMENDED to implement a default launcher that supports in-app pinning of shortcuts, widgets and widgetFeatures.
• [3.8.1/H-SR] Are STRONGLY RECOMMENDED to implement a default launcher that provides quick access to the additional shortcuts provided by third-party apps through the ShortcutManager API.
• [3.8.1/H-SR] Are STRONGLY RECOMMENDED to include a default launcher app that shows badges for the app icons.
• [3.8.2/H-SR] Are STRONGLY RECOMMENDED to support third-party app widgets.
• [3.8.3/H-0-1] MUST allow third-party apps to notify users of notable events through the and API classes.
• [3.8.3/H-0-2] MUST support rich notifications.
• [3.8.3/H-0-3] MUST support heads-up notifications.
• [3.8.3/H-0-4] MUST include a notification shade, providing the user the ability to directly control (e.g. reply, snooze, dismiss, block) the notifications through user affordance such as action buttons or the control panel as implemented in the AOSP.
• [3.8.3/H-0-5] MUST display the choices provided through in the notification shade.
• [3.8.3/H-SR] Are STRONGLY RECOMMENDED to display the first choice provided through in the notification shade without additional user interaction.
• [3.8.3/H-SR] Are STRONGLY RECOMMENDED to display all the choices provided through in the notification shade when the user expands all notifications in the notification shade.
• [3.8.3.1/H-SR] Are STRONGLY RECOMMENDED to display actions for which is set as in-line with the replies displayed by .
• [3.8.4/H-SR] Are STRONGLY RECOMMENDED to implement an assistant on the device to handle the Assist action.

If Handheld device implementations support Assist action, they:

• [3.8.4/H-SR] Are STRONGLY RECOMMENDED to use long press on key as the designated interaction to launch the assist app as described in section 7.2.3. MUST launch the user-selected assist app, in other words the app that implements , or an activity handling the intent.

If Handheld device implementations support and group them into a separate section from alerting and silent non-conversation notifications, they:

• [3.8.4/H-1-1]* MUST display conversation notifications ahead of non conversation notifications with the exception of ongoing foreground service notifications and importance:high notifications.

If Android Handheld device implementations support a lock screen, they:

• [3.8.10/H-1-1] MUST display the Lock screen Notifications including the Media Notification Template.

If Handheld device implementations support a secure lock screen, they:

• [3.9/H-1-1] MUST implement the full range of device administration policies defined in the Android SDK documentation.
• [3.9/H-1-2] MUST declare the support of managed profiles via the feature flag, except when the device is configured so that it would report itself as a low RAM device or so that it allocates internal (non-removable) storage as shared storage.

If Handheld device implementations include support for and APIs and allow third-party applications to publish , then they:

• [3.8.16/H-1-1] MUST declare the feature flag and set it to .
• [3.8.16/H-1-2] MUST provide a user affordance with the ability to add, edit, select, and operate the user’s favorite device controls from the controls registered by the third-party applications through the and the APIs.
• [3.8.16/H-1-3] MUST provide access to this user affordance within three interactions from a default Launcher.
• [3.8.16/H-1-4] MUST accurately render in this user affordance the name and icon of each third-party app that provides controls via the API as well as any specified fields provided by the APIs.

Conversely, If Handheld device implementations do not implement such controls, they:

Handheld device implementations:

• [3.10/H-0-1] MUST support third-party accessibility services.
• [3.10/H-SR] Are STRONGLY RECOMMENDED to preload accessibility services on the device comparable with or exceeding functionality of the Switch Access and TalkBack (for languages supported by the preinstalled Text-to-speech engine) accessibility services as provided in the talkback open source project.
• [3.11/H-0-1] MUST support installation of third-party TTS engines.
• [3.11/H-SR] Are STRONGLY RECOMMENDED to include a TTS engine supporting the languages available on the device.
• [3.13/H-SR] Are STRONGLY RECOMMENDED to include a Quick Settings UI component.

If Android handheld device implementations declare or support, they:

• [3.16/H-1-1] MUST support the companion device pairing feature.

If the navigation function is provided as an on-screen, gesture-based action:

• [7.2.3/H] The gesture recognition zone for the Home function SHOULD be no higher than 32 dp in height from the bottom of the screen.

If Handheld device implementations provide a navigation function as a gesture from anywhere on the left and right edges of the screen:

• [7.2.3/H-0-1] The navigation function's gesture area MUST be less than 40 dp in width on each side. The gesture area SHOULD be 24 dp in width by default.

2.2.4. Performance and Power

• [8.1/H-0-1] Consistent frame latency. Inconsistent frame latency or a delay to render frames MUST NOT happen more often than 5 frames in a second, and SHOULD be below 1 frames in a second.
• [8.1/H-0-2] User interface latency. Device implementations MUST ensure low latency user experience by scrolling a list of 10K list entries as defined by the Android Compatibility Test Suite (CTS) in less than 36 secs.
• [8.1/H-0-3] Task switching. When multiple applications have been launched, re-launching an already-running application after it has been launched MUST take less than 1 second.

Handheld device implementations:

• [8.2/H-0-1] MUST ensure a sequential write performance of at least 5 MB/s.
• [8.2/H-0-2] MUST ensure a random write performance of at least 0.5 MB/s.
• [8.2/H-0-3] MUST ensure a sequential read performance of at least 15 MB/s.
• [8.2/H-0-4] MUST ensure a random read performance of at least 3.5 MB/s.

If Handheld device implementations include features to improve device power management that are included in AOSP or extend the features that are included in AOSP, they:

• [8.3/H-1-1] MUST provide user affordance to enable and disable the battery saver feature.
• [8.3/H-1-2] MUST provide user affordance to display all apps that are exempted from App Standby and Doze power-saving modes.

Handheld device implementations:

• [8.4/H-0-1] MUST provide a per-component power profile that defines the current consumption value for each hardware component and the approximate battery drain caused by the components over time as documented in the Android Open Source Project site.
• [8.4/H-0-2] MUST report all power consumption values in milliampere hours (mAh).
• [8.4/H-0-3] MUST report CPU power consumption per each process's UID. The Android Open Source Project meets the requirement through the kernel module implementation.
• [8.4/H-0-4] MUST make this power usage available via the shell command to the app developer.
• [8.4/H] SHOULD be attributed to the hardware component itself if unable to attribute hardware component power usage to an application.

If Handheld device implementations include a screen or video output, they:

2.2.5. Security Model

Handheld device implementations:

• [9.1/H-0-1] MUST allow third-party apps to access the usage statistics via the permission and provide a user-accessible mechanism to grant or revoke access to such apps in response to the intent.

Handheld device implementations (* Not applicable for Tablet):

• [9.11/H-0-2]* MUST back up the keystore implementation with an isolated execution environment.
• [9.11/H-0-3]* MUST have implementations of RSA, AES, ECDSA, and HMAC cryptographic algorithms and MD5, SHA1, and SHA-2 family hash functions to properly support the Android Keystore system's supported algorithms in an area that is securely isolated from the code running on the kernel and above. Secure isolation MUST block all potential mechanisms by which kernel or userspace code might access the internal state of the isolated environment, including DMA. The upstream Android Open Source Project (AOSP) meets this requirement by using the Trusty implementation, but another ARM TrustZone-based solution or a third-party reviewed secure implementation of a proper hypervisor-based isolation are alternative options.
• [9.11/H-0-4]* MUST perform the lock screen authentication in the isolated execution environment and only when successful, allow the authentication-bound keys to be used. Lock screen credentials MUST be stored in a way that allows only the isolated execution environment to perform lock screen authentication. The upstream Android Open Source Project provides the Gatekeeper Hardware Abstraction Layer (HAL) and Trusty, which can be used to satisfy this requirement.
• [9.11/H-0-5]* MUST support key attestation where the attestation signing key is protected by secure hardware and signing is performed in secure hardware. The attestation signing keys MUST be shared across large enough number of devices to prevent the keys from being used as device identifiers. One way of meeting this requirement is to share the same attestation key unless at least 100,000 units of a given SKU are produced. If more than 100,000 units of an SKU are produced, a different key MAY be used for each 100,000 units.

Note that if a device implementation is already launched on an earlier Android version, such a device is exempted from the requirement to have a keystore backed by an isolated execution environment and support the key attestation, unless it declares the feature which requires a keystore backed by an isolated execution environment.

When Handheld device implementations support a secure lock screen, they:

• [9.11/H-1-1] MUST allow the user to choose the shortest sleep timeout, that is a transition time from the unlocked to the locked state, as 15 seconds or less.
• [9.11/H-1-2] MUST provide user affordance to hide notifications and disable all forms of authentication except for the primary authentication described in 9.11.1 Secure Lock Screen. The AOSP meets the requirement as lockdown mode.

2.2.6. Developer Tools and Options Compatibility

Handheld device implementations (* Not applicable for Tablet):

• [6.1/H-0-1]* MUST support the shell command .

Handheld device implementations (* Not applicable for Tablet):

• Perfetto
  • [6.1/H-0-2]* MUST expose a binary to the shell user which cmdline complies with the perfetto documentation.
  • [6.1/H-0-3]* The perfetto binary MUST accept as input a protobuf config that complies with the schema defined in the perfetto documentation.
  • [6.1/H-0-4]* The perfetto binary MUST write as output a protobuf trace that complies with the schema defined in the perfetto documentation.
  • [6.1/H-0-5]* MUST provide, through the perfetto binary, at least the data sources described in the perfetto documentation.
  • [6.1/H-0-6]* The perfetto traced daemon MUST be enabled by default (system property ).

2.3. Television Requirements

An Android Television device refers to an Android device implementation that is an entertainment interface for consuming digital media, movies, games, apps, and/or live TV for users sitting about ten feet away (a “lean back” or “10-foot user interface”).

Android device implementations are classified as a Television if they meet all the following criteria:

• Have provided a mechanism to remotely control the rendered user interface on the display that might sit ten feet away from the user.
• Have an embedded screen display with the diagonal length larger than 24 inches OR include a video output port, such as VGA, HDMI, DisplayPort, or a wireless port for display.

The additional requirements in the rest of this section are specific to Android Television device implementations.

2.3.1. Hardware

Television device implementations:

• [7.2.2/T-0-1] MUST support D-pad.
• [7.2.3/T-0-1] MUST provide the Home and Back functions.
• [7.2.3/T-0-2] MUST send both the normal and long press event of the Back function () to the foreground application.
• [7.2.6.1/T-0-1] MUST include support for game controllers and declare the feature flag.
• [7.2.7/T] SHOULD provide a remote control from which users can access non-touch navigation and core navigation keys inputs.

If Television device implementations include a 3-axis gyroscope, they:

• [7.3.4/T-1-1] MUST be able to report events up to a frequency of at least 100 Hz.
• [7.3.4/T-1-2] MUST be capable of measuring orientation changes up to 1000 degrees per second.

Television device implementations:

• [7.4.3/T-0-1] MUST support Bluetooth and Bluetooth LE.
• [7.6.1/T-0-1] MUST have at least 4 GB of non-volatile storage available for application private data (a.k.a. "/data" partition).

If Television device implementations include a USB port that supports host mode, they:

• [7.5.3/T-1-1] MUST include support for an external camera that connects through this USB port but is not necessarily always connected.

If TV device implementations are 32-bit:

• [7.6.1/T-1-1] The memory available to the kernel and userspace MUST be at least 896MB if any of the following densities are used:

  • 400dpi or higher on small/normal screens
  • xhdpi or higher on large screens
  • tvdpi or higher on extra large screens

If TV device implementations are 64-bit:

• [7.6.1/T-2-1] The memory available to the kernel and userspace MUST be at least 1280MB if any of the following densities are used:

  • 400dpi or higher on small/normal screens
  • xhdpi or higher on large screens
  • tvdpi or higher on extra large screens

Note that the "memory available to the kernel and userspace" above refers to the memory space provided in addition to any memory already dedicated to hardware components such as radio, video, and so on that are not under the kernel’s control on device implementations.

Television device implementations:

• [7.8.1/T] SHOULD include a microphone.
• [7.8.2/T-0-1] MUST have an audio output and declare .

2.3.2. Multimedia

Television device implementations MUST support the following audio encoding and decoding formats and make them available to third-party applications:

• [5.1/T-0-1] MPEG-4 AAC Profile (AAC LC)
• [5.1/T-0-2] MPEG-4 HE AAC Profile (AAC+)
• [5.1/T-0-3] AAC ELD (enhanced low delay AAC)

Television device implementations MUST support the following video encoding formats and make them available to third-party applications:

• [5.2/T-0-1] H.264
• [5.2/T-0-2] VP8

Television device implementations:

• [5.2.2/T-SR] Are STRONGLY RECOMMENDED to support H.264 encoding of 720p and 1080p resolution videos at 30 frames per second.

Television device implementations MUST support the following video decoding formats and make them available to third-party applications:

Television device implementations MUST support MPEG-2 decoding, as detailed in Section 5.3.1, at standard video frame rates and resolutions up to and including:

• [5.3.1/T-1-1] HD 1080p at 59.94 frames per second with Main Profile High Level.
• [5.3.1/T-1-2] HD 1080i at 59.94 frames per second with Main Profile High Level. They MUST deinterlace interlaced MPEG-2 video to its progressive equivalent (e.g. from 1080i at 59.94 frames per second to 1080p at 29.97 frames per second) and make it available to third-party applications.

Television device implementations MUST support H.264 decoding, as detailed in Section 5.3.4, at standard video frame rates and resolutions up to and including:

• [5.3.4/T-1-1] HD 1080p at 60 frames per second with Baseline Profile
• [5.3.4/T-1-2] HD 1080p at 60 frames per second with Main Profile
• [5.3.4/T-1-3] HD 1080p at 60 frames per second with High Profile Level 4.2

Television device implementations with H.265 hardware decoders MUST support H.265 decoding, as detailed in Section 5.3.5, at standard video frame rates and resolutions up to and including:

• [5.3.5/T-1-1] HD 1080p at 60 frames per second with Main Profile Level 4.1

If Television device implementations with H.265 hardware decoders support H.265 decoding and the UHD decoding profile, they:

• [5.3.5/T-2-1] MUST support UHD 3480p at 60 frames per second with Main10 Level 5 Main Tier profile

Television device implementations MUST support VP8 decoding, as detailed in Section 5.3.6, at standard video frame rates and resolutions up to and including:

• [5.3.6/T-1-1] HD 1080p at 60 frames per second decoding profile

Television device implementations with VP9 hardware decoders MUST support VP9 decoding, as detailed in Section 5.3.7, at standard video frame rates and resolutions up to and including:

• [5.3.7/T-1-1] HD 1080p at 60 frames per second with profile 0 (8 bit color depth)

If Television device implementations with VP9 hardware decoders support VP9 decoding and the UHD decoding profile, they:

• [5.3.7/T-2-1] MUST support UHD 3480p at 60 frames per second with profile 0 (8 bit color depth).
• [5.3.7/T-2-1] Are STRONGLY RECOMMENDED to support UHD 3480p at 60 frames per second with profile 2 (10 bit color depth).

Television device implementations:

• [5.5/T-0-1] MUST include support for system Master Volume and digital audio output volume attenuation on supported outputs, except for compressed audio passthrough output (where no audio decoding is done on the device).

If Television device implementations do not have a built in display, but instead support an external display connected via HDMI, they:

• [5.8/T-0-1] MUST set the HDMI output mode to select the maximum resolution that can be supported with either a 50Hz or 60Hz refresh rate.
• [5.8/T-SR] Are STRONGLY RECOMMENDED to provide a user configurable HDMI refresh rate selector.
• [5.8] SHOULD set the HDMI output mode refresh rate to either 50Hz or 60Hz, depending on the video refresh rate for the region the device is sold in.

If Television device implementations do not have a built in display, but instead support an external display connected via HDMI, they:

• [5.8/T-1-1] MUST support HDCP 2.2.

If Television device implementations do not support UHD decoding, but instead support an external display connected via HDMI, they:

• [5.8/T-2-1] MUST support HDCP 1.4

2.3.3. Software

Television device implementations:

• [3/T-0-1] MUST declare the features and .
• [3.2.3.1/T-0-1] MUST preload one or more applications or service components with an intent handler, for all the public intent filter patterns defined by the following application intents listed here.
• [3.4.1/T-0-1] MUST provide a complete implementation of the API.

If Android Television device implementations support a lock screen,they:

• [3.8.10/T-1-1] MUST display the Lock screen Notifications including the Media Notification Template.

Television device implementations:

• [3.8.14/T-SR] Are STRONGLY RECOMMENDED to support picture-in-picture (PIP) mode multi-window.
• [3.10/T-0-1] MUST support third-party accessibility services.
• [3.10/T-SR] Are STRONGLY RECOMMENDED to preload accessibility services on the device comparable with or exceeding functionality of the Switch Access and TalkBack (for languages supported by the preinstalled Text-to-speech engine) accessibility services as provided in the talkback open source project.

If Television device implementations report the feature , they:

• [3.11/T-SR] Are STRONGLY RECOMMENDED to include a TTS engine supporting the languages available on the device.
• [3.11/T-1-1] MUST support installation of third-party TTS engines.

Television device implementations:

• [3.12/T-0-1] MUST support TV Input Framework.

2.3.4. Performance and Power

• [8.1/T-0-1] Consistent frame latency. Inconsistent frame latency or a delay to render frames MUST NOT happen more often than 5 frames in a second, and SHOULD be below 1 frames in a second.
• [8.2/T-0-1] MUST ensure a sequential write performance of at least 5MB/s.
• [8.2/T-0-2] MUST ensure a random write performance of at least 0.5MB/s.
• [8.2/T-0-3] MUST ensure a sequential read performance of at least 15MB/s.
• [8.2/T-0-4] MUST ensure a random read performance of at least 3.5MB/s.

If Television device implementations include features to improve device power management that are included in AOSP or extend the features that are included in AOSP, they:

• [8.3/T-1-1] MUST provide user affordance to enable and disable the battery saver feature.

If Television device implementations do not have a battery they:

If Television device implementations have a battery they:

• [8.3/T-1-3] MUST provide user affordance to display all apps that are exempted from App Standby and Doze power-saving modes.

Television device implementations:

• [8.4/T-0-1] MUST provide a per-component power profile that defines the current consumption value for each hardware component and the approximate battery drain caused by the components over time as documented in the Android Open Source Project site.
• [8.4/T-0-2] MUST report all power consumption values in milliampere hours (mAh).
• [8.4/T-0-3] MUST report CPU power consumption per each process's UID. The Android Open Source Project meets the requirement through the kernel module implementation.
• [8.4/T] SHOULD be attributed to the hardware component itself if unable to attribute hardware component power usage to an application.
• [8.4/T-0-4] MUST make this power usage available via the shell command to the app developer.

2.3.5. Security Model

Television device implementations:

• [9.11/T-0-1] MUST back up the keystore implementation with an isolated execution environment.
• [9.11/T-0-2] MUST have implementations of RSA, AES, ECDSA and HMAC cryptographic algorithms and MD5, SHA1, and SHA-2 family hash functions to properly support the Android Keystore system's supported algorithms in an area that is securely isolated from the code running on the kernel and above. Secure isolation MUST block all potential mechanisms by which kernel or userspace code might access the internal state of the isolated environment, including DMA. The upstream Android Open Source Project (AOSP) meets this requirement by using the Trusty implementation, but another ARM TrustZone-based solution or a third-party reviewed secure implementation of a proper hypervisor-based isolation are alternative options.
• [9.11/T-0-3] MUST perform the lock screen authentication in the isolated execution environment and only when successful, allow the authentication-bound keys to be used. Lock screen credentials MUST be stored in a way that allows only the isolated execution environment to perform lock screen authentication. The upstream Android Open Source Project provides the Gatekeeper Hardware Abstraction Layer (HAL) and Trusty, which can be used to satisfy this requirement.
• [9.11/T-0-4] MUST support key attestation where the attestation signing key is protected by secure hardware and signing is performed in secure hardware. The attestation signing keys MUST be shared across large enough number of devices to prevent the keys from being used as device identifiers. One way of meeting this requirement is to share the same attestation key unless at least 100,000 units of a given SKU are produced. If more than 100,000 units of an SKU are produced, a different key MAY be used for each 100,000 units.

Note that if a device implementation is already launched on an earlier Android version, such a device is exempted from the requirement to have a keystore backed by an isolated execution environment and support the key attestation, unless it declares the feature which requires a keystore backed by an isolated execution environment.

If Television device implementations support a secure lock screen, they:

• [9.11/T-1-1] MUST allow the user to choose the Sleep timeout for transition from the unlocked to the locked state, with a minimum allowable timeout up to 15 seconds or less.

2.3.6. Developer Tools and Options Compatibility

Television device implementations:

An important feature of transactional databases like SQLite is "atomic commit". Atomic commit means that either all database changes within a single transaction occur or none of them occur. With atomic commit, it is as if many different writes to different sections of the database file occur instantaneously and simultaneously. Real hardware serializes writes to mass storage, and writing a single sector takes a finite amount of time. So it is impossible to truly write many different sectors of a database file simultaneously and/or instantaneously. But the atomic commit logic within SQLite makes it appear as if the changes for a transaction are all written instantaneously and simultaneously.

SQLite has the important property that transactions appear to be atomic even if the transaction is interrupted by an operating system crash or power failure.

This article describes the techniques used by SQLite to create the illusion of atomic commit.

The information in this article applies only when SQLite is operating in "rollback mode", or in other words when SQLite is not using a write-ahead log. SQLite still supports atomic commit when write-ahead logging is enabled, but it accomplishes atomic commit by a different mechanism from the one described in this article. See the write-ahead log documentation for additional information on how SQLite supports atomic commit in that context.

Throughout this article, we will call the mass storage device "disk" even though the mass storage device might really be flash memory.

We assume that disk is written in chunks which we call a "sector". It is not possible to modify any part of the disk smaller than a sector. To change a part of the disk smaller than a sector, you have to read in the full sector that contains the part you want to change, make the change, then write back out the complete sector.

On a traditional spinning disk, a sector is the minimum unit of transfer in both directions, both reading and writing. On flash memory, however, the minimum size of a read is typically much smaller than a minimum write. SQLite is only concerned with the minimum write amount and so for the purposes of this article, when we say "sector" we mean the minimum amount of data that can be written to mass storage in a single go.

Prior to SQLite version 3.3.14, a sector size of 512 bytes was assumed in all cases. There was a compile-time option to change this but the code had never been tested with a larger value. The 512 byte sector assumption seemed reasonable since until very recently all disk drives used a 512 byte sector internally. However, there has recently been a push to increase the sector size of disks to 4096 bytes. Also the sector size for flash memory is usually larger than 512 bytes. For these reasons, versions of SQLite beginning with 3.3.14 have a method in the OS interface layer that interrogates the underlying filesystem to find the true sector size. As currently implemented (version 3.5.0) this method still returns a hard-coded value of 512 bytes, since there is no standard way of discovering the true sector size on either Unix or Windows. But the method is available for embedded device manufacturers to tweak according to their own needs. And we have left open the possibility of filling in a more meaningful implementation on Unix and Windows in the future.

SQLite has traditionally assumed that a sector write is not atomic. However, SQLite does always assume that a sector write is linear. By "linear" we mean that SQLite assumes that when writing a sector, the hardware begins at one end of the data and writes byte by byte until it gets to the other end. The write might go from beginning to end or from end to beginning. If a power failure occurs in the middle of a sector write it might be that part of the sector was modified and another part was left unchanged. The key assumption by SQLite is that if any part of the sector gets changed, then either the first or the last bytes will be changed. So the hardware will never start writing a sector in the middle and work towards the ends. We do not know if this assumption is always true but it seems reasonable.

The previous paragraph states that SQLite does not assume that sector writes are atomic. This is true by default. But as of SQLite version 3.5.0, there is a new interface called the Virtual File System (VFS) interface. The VFS is the only means by which SQLite communicates to the underlying filesystem. The code comes with default VFS implementations for Unix and Windows and there is a mechanism for creating new custom VFS implementations at runtime. In this new VFS interface there is a method called xDeviceCharacteristics. This method interrogates the underlying filesystem to discover various properties and behaviors that the filesystem may or may not exhibit. The xDeviceCharacteristics method might indicate that sector writes are atomic, and if it does so indicate, SQLite will try to take advantage of that fact. But the default xDeviceCharacteristics method for both Unix and Windows does not indicate atomic sector writes and so these optimizations are normally omitted.

SQLite assumes that the operating system will buffer writes and that a write request will return before data has actually been stored in the mass storage device. SQLite further assumes that write operations will be reordered by the operating system. For this reason, SQLite does a "flush" or "fsync" operation at key points. SQLite assumes that the flush or fsync will not return until all pending write operations for the file that is being flushed have completed. We are told that the flush and fsync primitives are broken on some versions of Windows and Linux. This is unfortunate. It opens SQLite up to the possibility of database corruption following a power loss in the middle of a commit. However, there is nothing that SQLite can do to test for or remedy the situation. SQLite assumes that the operating system that it is running on works as advertised. If that is not quite the case, well then hopefully you will not lose power too often.

SQLite assumes that when a file grows in length that the new file space originally contains garbage and then later is filled in with the data actually written. In other words, SQLite assumes that the file size is updated before the file content. This is a pessimistic assumption and SQLite has to do some extra work to make sure that it does not cause database corruption if power is lost between the time when the file size is increased and when the new content is written. The xDeviceCharacteristics method of the VFS might indicate that the filesystem will always write the data before updating the file size. (This is the SQLITE_IOCAP_SAFE_APPEND property for those readers who are looking at the code.) When the xDeviceCharacteristics method indicates that files content is written before the file size is increased, SQLite can forego some of its pedantic database protection steps and thereby decrease the amount of disk I/O needed to perform a commit. The current implementation, however, makes no such assumptions for the default VFSes for Windows and Unix.

SQLite assumes that a file deletion is atomic from the point of view of a user process. By this we mean that if SQLite requests that a file be deleted and the power is lost during the delete operation, once power is restored either the file will exist completely with all if its original content unaltered, or else the file will not be seen in the filesystem at all. If after power is restored the file is only partially deleted, if some of its data has been altered or erased, or the file has been truncated but not completely removed, then database corruption will likely result.

SQLite assumes that the detection and/or correction of bit errors caused by cosmic rays, thermal noise, quantum fluctuations, device driver bugs, or other mechanisms, is the responsibility of the underlying hardware and operating system. SQLite does not add any redundancy to the database file for the purpose of detecting corruption or I/O errors. SQLite assumes that the data it reads is exactly the same data that it previously wrote.

By default, SQLite assumes that an operating system call to write a range of bytes will not damage or alter any bytes outside of that range even if a power loss or OS crash occurs during that write. We call this the "powersafe overwrite" property. Prior to version 3.7.9 (2011-11-01), SQLite did not assume powersafe overwrite. But with the standard sector size increasing from 512 to 4096 bytes on most disk drives, it has become necessary to assume powersafe overwrite in order to maintain historical performance levels and so powersafe overwrite is assumed by default in recent versions of SQLite. The assumption of powersafe overwrite property can be disabled at compile-time or a run-time if desired. See the powersafe overwrite documentation for further details.

We begin with an overview of the steps SQLite takes in order to perform an atomic commit of a transaction against a single database file. The details of file formats used to guard against damage from power failures and techniques for performing an atomic commit across multiple databases are discussed in later sections.

3.1. Initial State

The state of the computer when a database connection is first opened is shown conceptually by the diagram at the right. The area of the diagram on the extreme right (labeled "Disk") represents information stored on the mass storage device. Each rectangle is a sector. The blue color represents that the sectors contain original data. The middle area is the operating systems disk cache. At the onset of our example, the cache is cold and this is represented by leaving the rectangles of the disk cache empty. The left area of the diagram shows the content of memory for the process that is using SQLite. The database connection has just been opened and no information has been read yet, so the user space is empty.

3.2. Acquiring A Read Lock

Before SQLite can write to a database, it must first read the database to see what is there already. Even if it is just appending new data, SQLite still has to read in the database schema from the "sqlite_schema" table so that it can know how to parse the INSERT statements and discover where in the database file the new information should be stored.

The first step toward reading from the database file is obtaining a shared lock on the database file. A "shared" lock allows two or more database connections to read from the database file at the same time. But a shared lock prevents another database connection from writing to the database file while we are reading it. This is necessary because if another database connection were writing to the database file at the same time we are reading from the database file, we might read some data before the change and other data after the change. This would make it appear as if the change made by the other process is not atomic.

Notice that the shared lock is on the operating system disk cache, not on the disk itself. File locks really are just flags within the operating system kernel, usually. (The details depend on the specific OS layer interface.) Hence, the lock will instantly vanish if the operating system crashes or if there is a power loss. It is usually also the case that the lock will vanish if the process that created the lock exits.

3.3. Reading Information Out Of The Database

After the shared lock is acquired, we can begin reading information from the database file. In this scenario, we are assuming a cold cache, so information must first be read from mass storage into the operating system cache then transferred from operating system cache into user space. On subsequent reads, some or all of the information might already be found in the operating system cache and so only the transfer to user space would be required.

Usually only a subset of the pages in the database file are read. In this example we are showing three pages out of eight being read. In a typical application, a database will have thousands of pages and a query will normally only touch a small percentage of those pages.

3.4. Obtaining A Reserved Lock

Before making changes to the database, SQLite first obtains a "reserved" lock on the database file. A reserved lock is similar to a shared lock in that both a reserved lock and shared lock allow other processes to read from the database file. A single reserve lock can coexist with multiple shared locks from other processes. However, there can only be a single reserved lock on the database file. Hence only a single process can be attempting to write to the database at one time.

The idea behind a reserved lock is that it signals that a process intends to modify the database file in the near future but has not yet started to make the modifications. And because the modifications have not yet started, other processes can continue to read from the database. However, no other process should also begin trying to write to the database.

3.5. Creating A Rollback Journal File

Prior to making any changes to the database file, SQLite first creates a separate rollback journal file and writes into the rollback journal the original content of the database pages that are to be altered. The idea behind the rollback journal is that it contains all information needed to restore the database back to its original state.

The rollback journal contains a small header (shown in green in the diagram) that records the original size of the database file. So if a change causes the database file to grow, we will still know the original size of the database. The page number is stored together with each database page that is written into the rollback journal.

When a new file is created, most desktop operating systems (Windows, Linux, Mac OS X) will not actually write anything to disk. The new file is created in the operating systems disk cache only. The file is not created on mass storage until sometime later, when the operating system has a spare moment. This creates the impression to users that I/O is happening much faster than is possible when doing real disk I/O. We illustrate this idea in the diagram to the right by showing that the new rollback journal appears in the operating system disk cache only and not on the disk itself.

3.6. Changing Database Pages In User Space

After the original page content has been saved in the rollback journal, the pages can be modified in user memory. Each database connection has its own private copy of user space, so the changes that are made in user space are only visible to the database connection that is making the changes. Other database connections still see the information in operating system disk cache buffers which have not yet been changed. And so even though one process is busy modifying the database, other processes can continue to read their own copies of the original database content.

3.7. Flushing The Rollback Journal File To Mass Storage

The next step is to flush the content of the rollback journal file to nonvolatile storage. As we will see later, this is a critical step in insuring that the database can survive an unexpected power loss. This step also takes a lot of time, since writing to nonvolatile storage is normally a slow operation.

This step is usually more complicated than simply flushing the rollback journal to the disk. On most platforms two separate flush (or fsync()) operations are required. The first flush writes out the base rollback journal content. Then the header of the rollback journal is modified to show the number of pages in the rollback journal. Then the header is flushed to disk. The details on why we do this header modification and extra flush are provided in a later section of this paper.

3.8. Obtaining An Exclusive Lock

Prior to making changes to the database file itself, we must obtain an exclusive lock on the database file. Obtaining an exclusive lock is really a two-step process. First SQLite obtains a "pending" lock. Then it escalates the pending lock to an exclusive lock.

A pending lock allows other processes that already have a shared lock to continue reading the database file. But it prevents new shared locks from being established. The idea behind a pending lock is to prevent writer starvation caused by a large pool of readers. There might be dozens, even hundreds, of other processes trying to read the database file. Each process acquires a shared lock before it starts reading, reads what it needs, then releases the shared lock. If, however, there are many different processes all reading from the same database, it might happen that a new process always acquires its shared lock before the previous process releases its shared lock. And so there is never an instant when there are no shared locks on the database file and hence there is never an opportunity for the writer to seize the exclusive lock. A pending lock is designed to prevent that cycle by allowing existing shared locks to proceed but blocking new shared locks from being established. Eventually all shared locks will clear and the pending lock will then be able to escalate into an exclusive lock.

3.9. Writing Changes To The Database File

Once an exclusive lock is held, we know that no other processes are reading from the database file and it is safe to write changes into the database file. Usually those changes only go as far as the operating systems disk cache and do not make it all the way to mass storage.

3.10. 0 Flushing Changes To Mass Storage

Another flush must occur to make sure that all the database changes are written into nonvolatile storage. This is a critical step to ensure that the database will survive a power loss without damage. However, because of the inherent slowness of writing to disk or flash memory, this step together with the rollback journal file flush in section 3.7 above takes up most of the time required to complete a transaction commit in SQLite.

3.11. 1 Deleting The Rollback Journal

After the database changes are all safely on the mass storage device, the rollback journal file is deleted. This is the instant where the transaction commits. If a power failure or system crash occurs prior to this point, then recovery processes to be described later make it appear as if no changes were ever made to the database file. If a power failure or system crash occurs after the rollback journal is deleted, then it appears as if all changes have been written to disk. Thus, SQLite gives the appearance of having made no changes to the database file or having made the complete set of changes to the database file depending on whether or not the rollback journal file exists.

Deleting a file is not really an atomic operation, but it appears to be from the point of view of a user process. A process is always able to ask the operating system "does this file exist?" and the process will get back a yes or no answer. After a power failure that occurs during a transaction commit, SQLite will ask the operating system whether or not the rollback journal file exists. If the answer is "yes" then the transaction is incomplete and is rolled back. If the answer is "no" then it means the transaction did commit.

The existence of a transaction depends on whether or not the rollback journal file exists and the deletion of a file appears to be an atomic operation from the point of view of a user-space process. Therefore, a transaction appears to be an atomic operation.

The act of deleting a file is expensive on many systems. As an optimization, SQLite can be configured to truncate the journal file to zero bytes in length or overwrite the journal file header with zeros. In either case, the resulting journal file is no longer capable of rolling back and so the transaction still commits. Truncating a file to zero length, like deleting a file, is assumed to be an atomic operation from the point of view of a user process. Overwriting the header of the journal with zeros is not atomic, but if any part of the header is malformed the journal will not roll back. Hence, one can say that the commit occurs as soon as the header is sufficiently changed to make it invalid. Typically this happens as soon as the first byte of the header is zeroed.

3.12. 2 Releasing The Lock

The last step in the commit process is to release the exclusive lock so that other processes can once again start accessing the database file.

In the diagram at the right, we show that the information that was held in user space is cleared when the lock is released. This used to be literally true for older versions of SQLite. But more recent versions of SQLite keep the user space information in memory in case it might be needed again at the start of the next transaction. It is cheaper to reuse information that is already in local memory than to transfer the information back from the operating system disk cache or to read it off of the disk drive again. Prior to reusing the information in user space, we must first reacquire the shared lock and then we have to check to make sure that no other process modified the database file while we were not holding a lock. There is a counter in the first page of the database that is incremented every time the database file is modified. We can find out if another process has modified the database by checking that counter. If the database was modified, then the user space cache must be cleared and reread. But it is commonly the case that no changes have been made and the user space cache can be reused for a significant performance savings.

An atomic commit is supposed to happen instantaneously. But the processing described above clearly takes a finite amount of time. Suppose the power to the computer were cut part way through the commit operation described above. In order to maintain the illusion that the changes were instantaneous, we have to "rollback" any partial changes and restore the database to the state it was in prior to the beginning of the transaction.

4.1. When Something Goes Wrong...

Suppose the power loss occurred during step 3.10 above, while the database changes were being written to disk. After power is restored, the situation might be something like what is shown to the right. We were trying to change three pages of the database file but only one page was successfully written. Another page was partially written and a third page was not written at all.

The rollback journal is complete and intact on disk when the power is restored. This is a key point. The reason for the flush operation in step 3.7 is to make absolutely sure that all of the rollback journal is safely on nonvolatile storage prior to making any changes to the database file itself.

4.2. Hot Rollback Journals

The first time that any SQLite process attempts to access the database file, it obtains a shared lock as described in section 3.2 above. But then it notices that there is a rollback journal file present. SQLite then checks to see if the rollback journal is a "hot journal". A hot journal is a rollback journal that needs to be played back in order to restore the database to a sane state. A hot journal only exists when an earlier process was in the middle of committing a transaction when it crashed or lost power.

A rollback journal is a "hot" journal if all of the following are true:

• The rollback journal exists.
• The rollback journal is not an empty file.
• There is no reserved lock on the main database file.
• The header of the rollback journal is well-formed and in particular has not been zeroed out.
• The rollback journal does not contain the name of a super-journal file (see section 5.5 below) or if does contain the name of a super-journal, then that super-journal file exists.

The presence of a hot journal is our indication that a previous process was trying to commit a transaction but it aborted for some reason prior to the completion of the commit. A hot journal means that the database file is in an inconsistent state and needs to be repaired (by rollback) prior to being used.

4.3. Obtaining An Exclusive Lock On The Database

The first step toward dealing with a hot journal is to obtain an exclusive lock on the database file. This prevents two or more processes from trying to rollback the same hot journal at the same time.

4.4. Rolling Back Incomplete Changes

Once a process obtains an exclusive lock, it is permitted to write to the database file. It then proceeds to read the original content of pages out of the rollback journal and write that content back to where it came from in the database file. Recall that the header of the rollback journal records the original size of the database file prior to the start of the aborted transaction. SQLite uses this information to truncate the database file back to its original size in cases where the incomplete transaction caused the database to grow. At the end of this step, the database should be the same size and contain the same information as it did before the start of the aborted transaction.

4.5. Deleting The Hot Journal

After all information in the rollback journal has been played back into the database file (and flushed to disk in case we encounter yet another power failure), the hot rollback journal can be deleted.

As in section 3.11, the journal file might be truncated to zero length or its header might be overwritten with zeros as an optimization on systems where deleting a file is expensive. Either way, the journal is no longer hot after this step.

4.6. Continue As If The Uncompleted Writes Had Never Happened

The final recovery step is to reduce the exclusive lock back to a shared lock. Once this happens, the database is back in the state that it would have been if the aborted transaction had never started. Since all of this recovery activity happens completely automatically and transparently, it appears to the program using SQLite as if the aborted transaction had never begun.

SQLite allows a single database connection to talk to two or more database files simultaneously through the use of the ATTACH DATABASE command. When multiple database files are modified within a single transaction, all files are updated atomically. In other words, either all of the database files are updated or else none of them are. Achieving an atomic commit across multiple database files is more complex that doing so for a single file. This section describes how SQLite works that bit of magic.

5.1. Separate Rollback Journals For Each Database

When multiple database files are involved in a transaction, each database has its own rollback journal and each database is locked separately. The diagram at the right shows a scenario where three different database files have been modified within one transaction. The situation at this step is analogous to the single-file transaction scenario at step 3.6. Each database file has a reserved lock. For each database, the original content of pages that are being changed have been written into the rollback journal for that database, but the content of the journals have not yet been flushed to disk. No changes have been made to the database file itself yet, though presumably there are changes being held in user memory.

For brevity, the diagrams in this section are simplified from those that came before. Blue color still signifies original content and pink still signifies new content. But the individual pages in the rollback journal and the database file are not shown and we are not making the distinction between information in the operating system cache and information that is on disk. All of these factors still apply in a multi-file commit scenario. They just take up a lot of space in the diagrams and they do not add any new information, so they are omitted here.

5.2. The Super-Journal File

The next step in a multi-file commit is the creation of a "super-journal" file. The name of the super-journal file is the same name as the original database filename (the database that was opened using the sqlite3_open() interface, not one of the ATTACHed auxiliary databases) with the text "-mjHHHHHHHH" appended where HHHHHHHH is a random 32-bit hexadecimal number. The random HHHHHHHH suffix changes for every new super-journal.

(Nota bene: The formula for computing the super-journal filename given in the previous paragraph corresponds to the implementation as of SQLite version 3.5.0. But this formula is not part of the SQLite specification and is subject to change in future releases.)

Unlike the rollback journals, the super-journal does not contain any original database page content. Instead, the super-journal contains the full pathnames for rollback journals for every database that is participating in the transaction.

After the super-journal is constructed, its content is flushed to disk before any further actions are taken. On Unix, the directory that contains the super-journal is also synced in order to make sure the super-journal file will appear in the directory following a power failure.

The purpose of the super-journal is to ensure that multi-file transactions are atomic across a power-loss. But if the database files have other settings that compromise integrity across a power-loss event (such as PRAGMA synchronous=OFF or PRAGMA journal_mode=MEMORY) then the creation of the super-journal is omitted, as an optimization.

5.3. Updating Rollback Journal Headers

The next step is to record the full pathname of the super-journal file in the header of every rollback journal. Space to hold the super-journal filename was reserved at the beginning of each rollback journal as the rollback journals were created.

The content of each rollback journal is flushed to disk both before and after the super-journal filename is written into the rollback journal header. It is important to do both of these flushes. Fortunately, the second flush is usually inexpensive since typically only a single page of the journal file (the first page) has changed.

This step is analogous to step 3.7 in the single-file commit scenario described above.

5.4. Updating The Database Files

Once all rollback journal files have been flushed to disk, it is safe to begin updating database files. We have to obtain an exclusive lock on all database files before writing the changes. After all the changes are written, it is important to flush the changes to disk so that they will be preserved in the event of a power failure or operating system crash.

This step corresponds to steps 3.8, 3.9, and 3.10 in the single-file commit scenario described previously.

5.5. Delete The Super-Journal File

The next step is to delete the super-journal file. This is the point where the multi-file transaction commits. This step corresponds to step 3.11 in the single-file commit scenario where the rollback journal is deleted.

If a power failure or operating system crash occurs at this point, the transaction will not rollback when the system reboots even though there are rollback journals present. The difference is the super-journal pathname in the header of the rollback journal. Upon restart, SQLite only considers a journal to be hot and will only playback the journal if there is no super-journal filename in the header (which is the case for a single-file commit) or if the super-journal file still exists on disk.

5.6. Clean Up The Rollback Journals

The final step in a multi-file commit is to delete the individual rollback journals and drop the exclusive locks on the database files so that other processes can see the changes. This corresponds to step 3.12 in the single-file commit sequence.

The transaction has already committed at this point so timing is not critical in the deletion of the rollback journals. The current implementation deletes a single rollback journal then unlocks the corresponding database file before proceeding to the next rollback journal. But in the future we might change this so that all rollback journals are deleted before any database files are unlocked. As long as the rollback journal is deleted before its corresponding database file is unlocked it does not matter in what order the rollback journals are deleted or the database files are unlocked.

Section 3.0 above provides an overview of how atomic commit works in SQLite. But it glosses over a number of important details. The following subsections will attempt to fill in the gaps.

6.1. Always Journal Complete Sectors

When the original content of a database page is written into the rollback journal (as shown in section 3.5), SQLite always writes a complete sector of data, even if the page size of the database is smaller than the sector size. Historically, the sector size in SQLite has been hard coded to 512 bytes and since the minimum page size is also 512 bytes, this has never been an issue. But beginning with SQLite version 3.3.14, it is possible for SQLite to use mass storage devices with a sector size larger than 512 bytes. So, beginning with version 3.3.14, whenever any page within a sector is written into the journal file, all pages in that same sector are stored with it.

It is important to store all pages of a sector in the rollback journal in order to prevent database corruption following a power loss while writing the sector. Suppose that pages 1, 2, 3, and 4 are all stored in sector 1 and that page 2 is modified. In order to write the changes to page 2, the underlying hardware must also rewrite the content of pages 1, 3, and 4 since the hardware must write the complete sector. If this write operation is interrupted by a power outage, one or more of the pages 1, 3, or 4 might be left with incorrect data. Hence, to avoid lasting corruption to the database, the original content of all of those pages must be contained in the rollback journal.

6.2. Dealing With Garbage Written Into Journal Files

When data is appended to the end of the rollback journal, SQLite normally makes the pessimistic assumption that the file is first extended with invalid "garbage" data and that afterwards the correct data replaces the garbage. In other words, SQLite assumes that the file size is increased first and then afterwards the content is written into the file. If a power failure occurs after the file size has been increased but before the file content has been written, the rollback journal can be left containing garbage data. If after power is restored, another SQLite process sees the rollback journal containing the garbage data and tries to roll it back into the original database file, it might copy some of the garbage into the database file and thus corrupt the database file.

SQLite uses two defenses against this problem. In the first place, SQLite records the number of pages in the rollback journal in the header of the rollback journal. This number is initially zero. So during an attempt to rollback an incomplete (and possibly corrupt) rollback journal, the process doing the rollback will see that the journal contains zero pages and will thus make no changes to the database. Prior to a commit, the rollback journal is flushed to disk to ensure that all content has been synced to disk and there is no "garbage" left in the file, and only then is the page count in the header changed from zero to true number of pages in the rollback journal. The rollback journal header is always kept in a separate sector from any page data so that it can be overwritten and flushed without risking damage to a data page if a power outage occurs. Notice that the rollback journal is flushed to disk twice: once to write the page content and a second time to write the page count in the header.

The previous paragraph describes what happens when the synchronous pragma setting is "full".

PRAGMA synchronous=FULL;

The default synchronous setting is full so the above is what usually happens. However, if the synchronous setting is lowered to "normal", SQLite only flushes the rollback journal once, after the page count has been written. This carries a risk of corruption because it might happen that the modified (non-zero) page count reaches the disk surface before all of the data does. The data will have been written first, but SQLite assumes that the underlying filesystem can reorder write requests and that the page count can be burned into oxide first even though its write request occurred last. So as a second line of defense, SQLite also uses a 32-bit checksum on every page of data in the rollback journal. This checksum is evaluated for each page during rollback while rolling back a journal as described in section 4.4. If an incorrect checksum is seen, the rollback is abandoned. Note that the checksum does not guarantee that the page data is correct since there is a small but finite probability that the checksum might be right even if the data is corrupt. But the checksum does at least make such an error unlikely.

Note that the checksums in the rollback journal are not necessary if the synchronous setting is FULL. We only depend on the checksums when synchronous is lowered to NORMAL. Nevertheless, the checksums never hurt and so they are included in the rollback journal regardless of the synchronous setting.

6.3. Cache Spill Prior To Commit

The commit process shown in section 3.0 assumes that all database changes fit in memory until it is time to commit. This is the common case. But sometimes a larger change will overflow the user-space cache prior to transaction commit. In those cases, the cache must spill to the database before the transaction is complete.

At the beginning of a cache spill, the status of the database connection is as shown in step 3.6. Original page content has been saved in the rollback journal and modifications of the pages exist in user memory. To spill the cache, SQLite executes steps 3.7 through 3.9. In other words, the rollback journal is flushed to disk, an exclusive lock is acquired, and changes are written into the database. But the remaining steps are deferred until the transaction really commits. A new journal header is appended to the end of the rollback journal (in its own sector) and the exclusive database lock is retained, but otherwise processing returns to step 3.6. When the transaction commits, or if another cache spill occurs, steps 3.7 and 3.9 are repeated. (Step 3.8 is omitted on second and subsequent passes since an exclusive database lock is already held due to the first pass.)

A cache spill causes the lock on the database file to escalate from reserved to exclusive. This reduces concurrency. A cache spill also causes extra disk flush or fsync operations to occur and these operations are slow, hence a cache spill can seriously reduce performance. For these reasons a cache spill is avoided whenever possible.

Profiling indicates that for most systems and in most circumstances SQLite spends most of its time doing disk I/O. It follows then that anything we can do to reduce the amount of disk I/O will likely have a large positive impact on the performance of SQLite. This section describes some of the techniques used by SQLite to try to reduce the amount of disk I/O to a minimum while still preserving atomic commit.

7.1. Cache Retained Between Transactions

Step 3.12 of the commit process shows that once the shared lock has been released, all user-space cache images of database content must be discarded. This is done because without a shared lock, other processes are free to modify the database file content and so any user-space image of that content might become obsolete. Consequently, each new transaction would begin by rereading data which had previously been read. This is not as bad as it sounds at first since the data being read is still likely in the operating systems file cache. So the "read" is really just a copy of data from kernel space into user space. But even so, it still takes time.

Beginning with SQLite version 3.3.14 a mechanism has been added to try to reduce the needless rereading of data. In newer versions of SQLite, the data in the user-space pager cache is retained when the lock on the database file is released. Later, after the shared lock is acquired at the beginning of the next transaction, SQLite checks to see if any other process has modified the database file. If the database has been changed in any way since the lock was last released, the user-space cache is erased at that point. But commonly the database file is unchanged and the user-space cache can be retained, and some unnecessary read operations can be avoided.

In order to determine whether or not the database file has changed, SQLite uses a counter in the database header (in bytes 24 through 27) which is incremented during every change operation. SQLite saves a copy of this counter prior to releasing its database lock. Then after acquiring the next database lock it compares the saved counter value against the current counter value and erases the cache if the values are different, or reuses the cache if they are the same.

7.2. Exclusive Access Mode

SQLite version 3.3.14 adds the concept of "Exclusive Access Mode". In exclusive access mode, SQLite retains the exclusive database lock at the conclusion of each transaction. This prevents other processes from accessing the database, but in many deployments only a single process is using a database so this is not a serious problem. The advantage of exclusive access mode is that disk I/O can be reduced in three ways:

1. It is not necessary to increment the change counter in the database header for transactions after the first transaction. This will often save a write of page one to both the rollback journal and the main database file.

2. No other processes can change the database so there is never a need to check the change counter and clear the user-space cache at the beginning of a transaction.

3. Each transaction can be committed by overwriting the rollback journal header with zeros rather than deleting the journal file. This avoids having to modify the directory entry for the journal file and it avoids having to deallocate disk sectors associated with the journal. Furthermore, the next transaction will overwrite existing journal file content rather than append new content and on most systems overwriting is much faster than appending.

The third optimization, zeroing the journal file header rather than deleting the rollback journal file, does not depend on holding an exclusive lock at all times. This optimization can be set independently of exclusive lock mode using the journal_mode pragma as described in section 7.6 below.

7.3. Do Not Journal Freelist Pages

When information is deleted from an SQLite database, the pages used to hold the deleted information are added to a "freelist". Subsequent inserts will draw pages off of this freelist rather than expanding the database file.

Some freelist pages contain critical data; specifically the locations of other freelist pages. But most freelist pages contain nothing useful. These latter freelist pages are called "leaf" pages. We are free to modify the content of a leaf freelist page in the database without changing the meaning of the database in any way.

Because the content of leaf freelist pages is unimportant, SQLite avoids storing leaf freelist page content in the rollback journal in step 3.5 of the commit process. If a leaf freelist page is changed and that change does not get rolled back during a transaction recovery, the database is not harmed by the omission. Similarly, the content of a new freelist page is never written back into the database at step 3.9 nor read from the database at step 3.3. These optimizations can greatly reduce the amount of I/O that occurs when making changes to a database file that contains free space.

7.4. Single Page Updates And Atomic Sector Writes

Beginning in SQLite version 3.5.0, the new Virtual File System (VFS) interface contains a method named xDeviceCharacteristics which reports on special properties that the underlying mass storage device might have. Among the special properties that xDeviceCharacteristics might report is the ability of to do an atomic sector write.

Recall that by default SQLite assumes that sector writes are linear but not atomic. A linear write starts at one end of the sector and changes information byte by byte until it gets to the other end of the sector. If a power loss occurs in the middle of a linear write then part of the sector might be modified while the other end is unchanged. In an atomic sector write, either the entire sector is overwritten or else nothing in the sector is changed.

We believe that most modern disk drives implement atomic sector writes. When power is lost, the drive uses energy stored in capacitors and/or the angular momentum of the disk platter to provide power to complete any operation in progress. Nevertheless, there are so many layers in between the write system call and the on-board disk drive electronics that we take the safe approach in both Unix and w32 VFS implementations and assume that sector writes are not atomic. On the other hand, device manufacturers with more control over their filesystems might want to consider enabling the atomic write property of xDeviceCharacteristics if their hardware really does do atomic writes.

When sector writes are atomic and the page size of a database is the same as a sector size, and when there is a database change that only touches a single database page, then SQLite skips the whole journaling and syncing process and simply writes the modified page directly into the database file. The change counter in the first page of the database file is modified separately since no harm is done if power is lost before the change counter can be updated.

7.5. Filesystems With Safe Append Semantics

Another optimization introduced in SQLite version 3.5.0 makes use of "safe append" behavior of the underlying disk. Recall that SQLite assumes that when data is appended to a file (specifically to the rollback journal) that the size of the file is increased first and that the content is written second. So if power is lost after the file size is increased but before the content is written, the file is left containing invalid "garbage" data. The xDeviceCharacteristics method of the VFS might, however, indicate that the filesystem implements "safe append" semantics. This means that the content is written before the file size is increased so that it is impossible for garbage to be introduced into the rollback journal by a power loss or system crash.

When safe append semantics are indicated for a filesystem, SQLite always stores the special value of -1 for the page count in the header of the rollback journal. The -1 page count value tells any process attempting to rollback the journal that the number of pages in the journal should be computed from the journal size. This -1 value is never changed. So that when a commit occurs, we save a single flush operation and a sector write of the first page of the journal file. Furthermore, when a cache spill occurs we no longer need to append a new journal header to the end of the journal; we can simply continue appending new pages to the end of the existing journal.

7.6. Persistent Rollback Journals

Deleting a file is an expensive operation on many systems. So as an optimization, SQLite can be configured to avoid the delete operation of section 3.11. Instead of deleting the journal file in order to commit a transaction, the file is either truncated to zero bytes in length or its header is overwritten with zeros. Truncating the file to zero length saves having to make modifications to the directory containing the file since the file is not removed from the directory. Overwriting the header has the additional savings of not having to update the length of the file (in the "inode" on many systems) and not having to deal with newly freed disk sectors. Furthermore, at the next transaction the journal will be created by overwriting existing content rather than appending new content onto the end of a file, and overwriting is often much faster than appending.

SQLite can be configured to commit transactions by overwriting the journal header with zeros instead of deleting the journal file by setting the "PERSIST" journaling mode using the journal_mode PRAGMA. For example:

PRAGMA journal_mode=PERSIST;

The use of persistent journal mode provides a noticeable performance improvement on many systems. Of course, the drawback is that the journal files remain on the disk, using disk space and cluttering directories, long after the transaction commits. The only safe way to delete a persistent journal file is to commit a transaction with journaling mode set to DELETE:

PRAGMA journal_mode=DELETE; BEGIN EXCLUSIVE; COMMIT;

Beware of deleting persistent journal files by any other means since the journal file might be hot, in which case deleting it will corrupt the corresponding database file.

Beginning in SQLite version 3.6.4 (2008-10-15), the TRUNCATE journal mode is also supported:

PRAGMA journal_mode=TRUNCATE;

In truncate journal mode, the transaction is committed by truncating the journal file to zero length rather than deleting the journal file (as in DELETE mode) or by zeroing the header (as in PERSIST mode). TRUNCATE mode shares the advantage of PERSIST mode that the directory that contains the journal file and database does not need to be updated. Hence truncating a file is often faster than deleting it. TRUNCATE has the additional advantage that it is not followed by a system call (ex: fsync()) to synchronize the change to disk. It might be safer if it did. But on many modern filesystems, a truncate is an atomic and synchronous operation and so we think that TRUNCATE will usually be safe in the face of power failures. If you are uncertain about whether or not TRUNCATE will be synchronous and atomic on your filesystem and it is important to you that your database survive a power loss or operating system crash that occurs during the truncation operation, then you might consider using a different journaling mode.

On embedded systems with synchronous filesystems, TRUNCATE results in slower behavior than PERSIST. The commit operation is the same speed. But subsequent transactions are slower following a TRUNCATE because it is faster to overwrite existing content than to append to the end of a file. New journal file entries will always be appended following a TRUNCATE but will usually overwrite with PERSIST.

The developers of SQLite are confident that it is robust in the face of power failures and system crashes because the automatic test procedures do extensive checks on the ability of SQLite to recover from simulated power loss. We call these the "crash tests".

Crash tests in SQLite use a modified VFS that can simulate the kinds of filesystem damage that occur during a power loss or operating system crash. The crash-test VFS can simulate incomplete sector writes, pages filled with garbage data because a write has not completed, and out of order writes, all occurring at varying points during a test scenario. Crash tests execute transactions over and over, varying the time at which a simulated power loss occurs and the properties of the damage inflicted. Each test then reopens the database after the simulated crash and verifies that the transaction either occurred completely or not at all and that the database is in a completely consistent state.

The crash tests in SQLite have discovered a number of very subtle bugs (now fixed) in the recovery mechanism. Some of these bugs were very obscure and unlikely to have been found using only code inspection and analysis techniques. From this experience, the developers of SQLite feel confident that any other database system that does not use a similar crash test system likely contains undetected bugs that will lead to database corruption following a system crash or power failure.

The atomic commit mechanism in SQLite has proven to be robust, but it can be circumvented by a sufficiently creative adversary or a sufficiently broken operating system implementation. This section describes a few of the ways in which an SQLite database might be corrupted by a power failure or system crash. (See also: How To Corrupt Your Database Files.)

9.1. Broken Locking Implementations

SQLite uses filesystem locks to make sure that only one process and database connection is trying to modify the database at a time. The filesystem locking mechanism is implemented in the VFS layer and is different for every operating system. SQLite depends on this implementation being correct. If something goes wrong and two or more processes are able to write the same database file at the same time, severe damage can result.

We have received reports of implementations of both Windows network filesystems and NFS in which locking was subtly broken. We can not verify these reports, but as locking is difficult to get right on a network filesystem we have no reason to doubt them. You are advised to avoid using SQLite on a network filesystem in the first place, since performance will be slow. But if you must use a network filesystem to store SQLite database files, consider using a secondary locking mechanism to prevent simultaneous writes to the same database even if the native filesystem locking mechanism malfunctions.

The versions of SQLite that come preinstalled on Apple Mac OS X computers contain a version of SQLite that has been extended to use alternative locking strategies that work on all network filesystems that Apple supports. These extensions used by Apple work great as long as all processes are accessing the database file in the same way. Unfortunately, the locking mechanisms do not exclude one another, so if one process is accessing a file using (for example) AFP locking and another process (perhaps on a different machine) is using dot-file locks, the two processes might collide because AFP locks do not exclude dot-file locks or vice versa.

9.2. Incomplete Disk Flushes

SQLite uses the fsync() system call on Unix and the FlushFileBuffers() system call on w32 in order to sync the file system buffers onto disk oxide as shown in step 3.7 and step 3.10. Unfortunately, we have received reports that neither of these interfaces works as advertised on many systems. We hear that FlushFileBuffers() can be completely disabled using registry settings on some Windows versions. Some historical versions of Linux contain versions of fsync() which are no-ops on some filesystems, we are told. Even on systems where FlushFileBuffers() and fsync() are said to be working, often the IDE disk control lies and says that data has reached oxide while it is still held only in the volatile control cache.

On the Mac, you can set this pragma:

PRAGMA fullfsync=ON;

Setting fullfsync on a Mac will guarantee that data really does get pushed out to the disk platter on a flush. But the implementation of fullfsync involves resetting the disk controller. And so not only is it profoundly slow, it also slows down other unrelated disk I/O. So its use is not recommended.

9.3. Partial File Deletions

SQLite assumes that file deletion is an atomic operation from the point of view of a user process. If power fails in the middle of a file deletion, then after power is restored SQLite expects to see either the entire file with all of its original data intact, or it expects not to find the file at all. Transactions may not be atomic on systems that do not work this way.

9.4. Garbage Written Into Files

SQLite database files are ordinary disk files that can be opened and written by ordinary user processes. A rogue process can open an SQLite database and fill it with corrupt data. Corrupt data might also be introduced into an SQLite database by bugs in the operating system or disk controller; especially bugs triggered by a power failure. There is nothing SQLite can do to defend against these kinds of problems.

9.5. Deleting Or Renaming A Hot Journal

If a crash or power loss does occur and a hot journal is left on the disk, it is essential that the original database file and the hot journal remain on disk with their original names until the database file is opened by another SQLite process and rolled back. During recovery at step 4.2 SQLite locates the hot journal by looking for a file in the same directory as the database being opened and whose name is derived from the name of the file being opened. If either the original database file or the hot journal have been moved or renamed, then the hot journal will not be seen and the database will not be rolled back.

We suspect that a common failure mode for SQLite recovery happens like this: A power failure occurs. After power is restored, a well-meaning user or system administrator begins looking around on the disk for damage. They see their database file named "important.data". This file is perhaps familiar to them. But after the crash, there is also a hot journal named "important.data-journal". The user then deletes the hot journal, thinking that they are helping to cleanup the system. We know of no way to prevent this other than user education.

If there are multiple (hard or symbolic) links to a database file, the journal will be created using the name of the link through which the file was opened. If a crash occurs and the database is opened again using a different link, the hot journal will not be located and no rollback will occur.

Sometimes a power failure will cause a filesystem to be corrupted such that recently changed filenames are forgotten and the file is moved into a "/lost+found" directory. When that happens, the hot journal will not be found and recovery will not occur. SQLite tries to prevent this by opening and syncing the directory containing the rollback journal at the same time it syncs the journal file itself. However, the movement of files into /lost+found can be caused by unrelated processes creating unrelated files in the same directory as the main database file. And since this is out from under the control of SQLite, there is nothing that SQLite can do to prevent it. If you are running on a system that is vulnerable to this kind of filesystem namespace corruption (most modern journalling filesystems are immune, we believe) then you might want to consider putting each SQLite database file in its own private subdirectory.

Every now and then someone discovers a new failure mode for the atomic commit mechanism in SQLite and the developers have to put in a patch. This is happening less and less and the failure modes are becoming more and more obscure. But it would still be foolish to suppose that the atomic commit logic of SQLite is entirely bug-free. The developers are committed to fixing these bugs as quickly as they might be found.

The developers are also on the lookout for new ways to optimize the commit mechanism. The current VFS implementations for Unix (Linux and Mac OS X) and Windows make pessimistic assumptions about the behavior of those systems. After consultation with experts on how these systems work, we might be able to relax some of the assumptions on these systems and allow them to run faster. In particular, we suspect that most modern filesystems exhibit the safe append property and that many of them might support atomic sector writes. But until this is known for certain, SQLite will take the conservative approach and assume the worst.