Foreword

Welcome!

Welcome to NightOS' documentation! This book will show you how the system is designed, how it works, and all the little things it can offer to the end user as well as the features availables for application developers.

Before reading this documentation, please note that I'm developing this project on my own and I'm by no mean an operating system expert. I do not even as a professional low-level developer though I have some knowledge on how low-level software and hardware work to some extent.

This means there are surely many flaws in the very low-level aspects, especially in the microkernel, like the way hardware devices are handled and mapped in memory. If you find a flaw, feel free to fill an issue.

The main point of this project is the many middle-level/high-level aspects, not the very low-level ones which will take quite a lot of time to design completely and surely require the knowledge of people more experimented in this field than I am.

On the other hand, specification documents are not like specification documents you may have seen from, let's say, WhatWG or other consortiums. As NightOS is only a prototypal O.S., the documents you will find here are meant to be easily understandable. Specification documents are meant to describe completely a specific concept or part of the system, but without being hard to read. Some will argue it's not close enough to a "real" specification, but I think it's enough for now, regarding the very early state of the project.

Also, this project is far from being finished, so many things are still missing from the documentation. I frequently add new design documents and complete existing ones, but feel free to create a new issue if you think something should be added to the project :)

By the way, you can also find answers to the most frequently-asked questions here.

What is NightOS?

NightOS is an operating system aiming to replace ancient systems like Windows, Linux or MacOS. Well, this is the ideal goal but given how much Windows/Linux/MacOS and other systems are deeply installed in today's computers, NightOS is more of a theorical operating system that shows what we could get if we hadn't to maintain legacy compatibility with ancient architectures.

What? Another O.S.? Why?

The problem with current OSes like Windows, Linux or MacOS is they were built on bad roots. At the time they were created, security wasn't a concern like today, and computers weren't nearly as powerful as they are now. Which means a lot of things we have to deal with today in order to maintain a good level of security or performance isn't met at all with these systems, because they can't just be rewritten from scratch or break compatibility with older pieces of software.

NightOS, on its side, starts from a blank page. There is no compatibility to maintain on the software level, and everything is designed from the start to be future-proof, meaning it will be easy to add new security features for instance, without breaking any software compatibility.

What's the project's structure?

The project is composed of three distinct parts: the kernel, the system, and the desktop environment.

The first one is the part that allows the software to access the hardware, like writing files to the disk or making network requests. It's a microkernel, meaning it's easier to maintain and has a reduced attack surface for potential security issues than a monolithic kernel like Linux. In its early stages though, we will use a the Linux kernel as a base, and the microkernel will be last "brick" to the project.

The system is the higher-level component that deals with everything that is not directly related to hardware or graphical interface. For instance, applications, user accounts and permissions are all managed by the system. It's a read-only part the user cannot modify by itself.

The desktop environment, finally, is the part the user sees when launching NightOS. It noteably contains the status bar, the windows manager, the notifications panel, and all graphical-related things.

The goal of NightOS is to provide an operating system that is far more secure, robust and performant than existing systems, and this for all users.

What license is this project distributed under?

The NightOS project uses the Apache 2.0 license.

Special thanks

Thanks to rmRander for providing the image that served as a base for this project's logo. You can find the original picture here.

Thanks to Render for hosting this documentation.

Frequently-Asked Questions

Welcome to NightOS' FAQ. This document answers to common questions about the project, completing the foreword.

• This project seems too good to be true
• What's the current state of the project?
• Will this project replace Windows/MacOS/Linux/... one day?
• How does this project relates to NightOS v1, v2 and v3?
• Who are you?
• Why did you create NightOS?
• How can I help?

This project seems too good to be true

Well, it is in a way. NightOS can be designed as such a secure and robust O.S. because we chose to not maintain any backward compatibility with existing software - understand that existing applications from Linux, MacOS or Windows will not be compatible with NightOS. This is why we can afford to introduce so many new features and requirements current operating systems cannot.

The problem is that backward compatibility is absolutely mandatory when it comes to creating a general-purpose operating system. All applications would have to be developed again (in fact, they would just have to be adapted for NightOS, but that's still quite a bit of work) in order to run on this system, which is of course unfeasable today.

This is why NightOS will probably never be completed. The current goal of this project is to build a theorical O.S. that demonstrates how good of an O.S. we could make if we hadn't to worry about backward compatibility. When the theorical specifications will be complete, we shall start to develop the O.S. itself but that will require a lot of work beforehand.

Still, we do all we can to maintain as much compatibility as we can. For instance, we are currently investigating how existing Linux applications may be ran on NightOS, with a system call wrapper that would allow them to run correctly on the system, without compromising security, but with lacking features of course.

What's the current state of the project?

The project is currently in its very early stages. You can find more informations about this in the ROADMAP.

Will this project replace Windows/MacOS/Linux/... one day?

No. Absolutely not. Because, as I said in sooner, it's not viable for an operating system today to just get rid of all backward compatibility with existing software, simply because what users are interested in is applications first, and then the system itself - is it secure, stable, performant, and so on.

Imagine someone introducing the perfect operating system, but with almost no application on it. Would you get rid of your Windows/MacOS/Linux/... system for this one? Surely not, at least not as your everyday system.

This is why this project is for now purely theorical. It aims to describe how great an OS could be if we did not have to care about backward compatibility. Changes to see it in a usable state one day are like very low.

How does this project relates to NightOS v1, v2 and v3?

The current NightOS project is very different from its previous versions (v1, v2 and v3). Originally, it only intended to be a robust and secure desktop environment, written in JavaScript. Performances would of course be very bad, but it was more of a challenge than a serious project. Also, the project was meant to be a Linux kernel under the hood to get rid of the "low-level part". I seriously lacked of knowledge about low-level concepts at this point in time.

This project does not use JavaScript anymore, but Rust, which is both performant and very robust to the most common memory-related bugs (unless you use unsafe code). There is no more "challenge" idea involved in this new version, only the aim to make the best possible (theorical) system.

In terms of similarities:

• The v1 was more of a draft, which had a lot of problems up to its roots. It was a pretty bad version overall, and was created when I did not have a real knowledge about how an operating system actually worked ;
• The v2 was a bit more realistic, but still far from being mature enough ;
• The v3 was more serious and ambitious, but it was still a desktop environment and not designed to be a full operating system, plus it was not designed for optimal performances

Who are you?

My name is Clément Nerma (my last name is a pseudonym). I'm a back-end developer that makes stuff since I'm 10-year old and I love to touch to low-level concepts, like operating systems.

Why did you create NightOS?

I created it as an answer to the frustration modern OSes provided me. All modern systems have many stability and security problems that cannot be resolved easily because they need to maintain backward compatibility with older pieces of software. This is where I had the idea of creating NightOS: an operating system that is both robust (understand stable, with mechanisms to minimize data loss in case something goes wrong) and secure, with many security features available to the average user and some more complex ones for advanced users.

How can I help?

Help is very welcome but there's not many things to help with currently, as the system is still being designed. You can still help by suggesting improvements or fixing an error (whatever it is, a design problem or a simple typo) by submitting an issue.

Project

This folder contains the documentation about the project itself - how is will be developed, maintained, its release schedule, etc. Please note that this part is far from being complete, and many informations will be missing and/or incomplete.

• Roadmap - the project's roadmap
• Development - how the project will be developped
• Hardware requirements - hardware required in order to install and run NightOS

Project's ROADMAP

WARNING: This document is only a draft and as such far from being complete. All informations described here are subject to change anytime.

The project will be cut in the four phases described below.

• Conception
• Validation
• Implementation
  • Phase 1: Unoptimized kernel
  • Phase 2: Low-level implementation
  • Phase 3: Applications & System optimization
  • Phase 4: Completion
• Evolution, Optimization & Maintenance

Conception

The first part is to write conception documents about how the system work, both at a high-level (concepts, features, native applications etc.) and low-level (processes management, kernel design...).

This project is currently in this state.

Validation

Once all conception documents are ready, they will be validated one by one, and frozen. Breaking changes after validation are still possible but should be avoided as much as possible.

Implementation

Once all documents have been validated, the different project's pieces can be implemented.

It may happen that, during implementation, some validated documents get modifications in order to improve or fix some specific points. Each modification will need to be validated in order to preserve the stability of these documents.

Below are described the different phases of the implementation.

Phase 1: Unoptimized kernel

The first piece to be implemented will be the kernel. This is required as the kernel is needed in order to implement higher-level pieces like processes. Still, as not everything is needed right from the start, it will developed with no special matter of performances. In this state, it will be referred to as the name of unoptimized kernel.

Phase 2: Low-level implementation

Once the unoptimized kernel is ready, two implementations will start in parallel:

• Kernel optimization (as the unoptimized kernel will be really, really slow)
• System development

The system development will consist in building the following pieces successively:

1. The BIOS/UEFI bootloader
2. The system bootloader
3. The process manager
4. The native process manager

Phase 3: Applications & System optimization

The third phase consists in developing the native applications, which are meant to be used by the end user, as well as optimizing the system in parallel and improving its stability.

Starting from this point, all conception documents are definitively validated, which means they cannot receive breaking changes anymore, only new points/features.

This also coincides with the system's first alpha release, which will allow persons who are external to the project to test it and suggest improvements.

Phase 4: Completion

The last phase's purpose is to ensure the system is in a fully usable state, in other words to stabilize it.

The kernel and system will also need to be properly optimized, and once all these things are done a first beta release will be deployed.

Evolution, Optimization & Maintenance

The Evolution, Optimization & Maintenance (EPM) part is pretty explicit: it's all about improving NightOS' existing features, introducing new ones as needed, improving performances and stability, and fixing new bugs.

This part will last forever, as for all operating systems - or until the project dies.

Project development

WARNING: This document is only a draft and as such far from being complete. All informations described here are subject to change anytime.

• Languages

Languages

The project will be developped in Rust, with first-class support for this language. API interopability will be assured for TypeScript in the future.

Usage of architecture-specific assembly will be reduced as much as possible, being only used in two cases:

• The desired behaviour cannot be reached without assembly (e.g. direct register or stack manipulation)
• Extremely performances-critical pieces

Hardware requirements

WARNING: This document is only a draft and as such far from being complete. All informations described here are subject to change anytime.

This document presents the minimal hardware requirements required by NightOS in order to work. This is also means NightOS will work on any computer with these attributes.

• Processor: 64-bit amd64 processor
• Memory: at least 256 MB or RAM
• Storage: to be determined

Recommanded configuration

• A modern process with a built-in cryptography module
• An I/O Memory Management Unit (IOMMU) for DMA access

Concepts

This folder contains documentation for each high-level concept of NightOS. They are designed to be easily readable and understandable.

• What are applications? - the way to run software on NightOS
• What are libraries? - sharing identical behaviours between multiple applications
• What are users? - sharing a computer between multiple persons

Applications

Applications are the system's way to handle software.

NOTE : This document is only an introduction to how applications work.

• How applications work
• Installation methods
  • From the store
  • Sideloading
  • Volatile applications
• Permissions
• Name and slug
• Application Identifier
• Application Context
• Commands
• System applications
• Services

How applications work

An application is a set of executable files and resources. They are the only way to execute code, as direct binary programs are not supported.

Any user can install applications, which will only be available from his account. Administrator users can install global applications, which are available to every user.

Installation methods

Applications are installed through an application package via Skyer, the applications manager. There are several installation methods:

• From the store
• Directly from the application's package (sideloading)
• As a volatile application

From the store

Applications can be downloaded from NightOS' official applications store (available via Stellar).

• For closed-source applications, the store only provides pre-compiled programs
• For open-source applications, the store provides both pre-built programs as well as the source code

For the latter, user can choose either to build the program from source, using the appropriated build tools in order to optimize performances, or to simply use the pre-built programs (which is the option by default).

Sideloading

Applications sideloading (installing an application directly from its package) follows strict rules determined by the sideloading mode, which is either "disabled", "secure" or "unsecure".

Disable mode prevents all sideloading ; it's not possible to install applications from their package in this mode. Volatile applications can stil be run, though.

Secure mode allows sideloading but will first make the system check if the application's AID matches an existing application on the Store. If so, it compares the application's signature to the Store application's one. If they don't match, the application is considered malicious and won't be installed.
Note that this mode only works while connected to internet, as the system needs to check the Store to ensure the application is not malicious. If the computer is offline, sideloading will be disabled.

Unsecure mode allows sideloading without any checking, which is highly dangerous as it allows spoofing.

The sideloading mode can be changed in the control center.

Volatile applications

Applications can be also be ran as volatile applications, which means they are not installed on the disk. There are three methods:

• Full-volatile: the app's data are removed when the application closes
• Session-scoped: the app's data are stored on disk until the system shuts down
• Local-persistent: the app's data are stored within a data file located in the same folder
• Persistent: the app's data are stored in a dedicated folder, also enabling common data between users

By default, volatile applications are ran in local-persistent mode. In this mode, the system first checks if a file with the same name as the application's package but with the .vad (Volatile Application's Data) exists. If so, it opens this file as the application's storage. Then, when the application wants to store some data, it is stored inside this data file.

Note that VAD files are disguised VST files.

Volatile applications running as persistent do not appear in the applications list and can only be managed through a specific option in the Control Center. Their executable files are not stored anywhere and stay in the application's package, while only their data are stored on the disk. This allows to run the same application several times without losing any data and without worrying about a data file. This also allows to store common data between users.

Note that the store has an option for installing applications as volatile.

Permissions

See the permissions feature document.

Name and slug

Each application has a name as well as a slug. The name can any valid UTF-8 string, while the slug must respect several rules:

• Only lowercase letters, underscores and digits
• Must not start by a digit
• Must not be the name of a native shell command
• Must not be the name of a native shell function
• Must not be the name of a shell type

By default, the slug is auto-generated from the name, but it can also be customized.

Application Identifier

From the slug is generated the Application's IDentifier (AID), which is prefixed by the developer's identifier (DID) specified in the application's manifest (it must match the publisher's identifier on the store), and followed by two double points. The DID is submitted to the same rules as the application's slug.

For instance, an application with a slug of utils made by a developer whose DID is superdev will get an AID of utils::superdev.

The AID is unique across the store as well as in a single NightOS installation. As malicious application may provide the DID and the slug of a legit application (which is called AID spoofing), sideloading is verified by default.

As AID are text-based and can be quite long (up to 512 bytes), programs can instead use the Application's Numeric IDentifier (ANID), which is a 32-bit unique number randomly generated by the system to refer to this particular application. On two different systems, the ANID of a given application will likely be very different, and so cannot be guessed. It is provided by the system during the application's launch through the context.

System applications are registered under the reserved sys DID.

Application Context

When an application starts, it can retrieve its context, which are data indicating the execution context of the application. Detailed informations can be found in the related specifications document.

Commands

Application can expose shell commands. Multiple commands can be exposed without any risk of clashing as the command name must be prefixed by the AID first.

For instance, if an application with AID superdev.utils exposes an get_time command, the final usable command will be :superdev.utils.get_time.

This is quite a long name but allows to prevent any clashing between commands. It's common for shell scripts to use imports at the beginning of the script to refer more easily to applications' commands.

Note that, by default, shell prompts (not scripts) will allow to directly use commands such as get_time in the short form if no other application exposes a command with the same name.

Commands work by launching the application with a specific context.

System applications

Some native applications are part of the system itself and are called system applications as such. They get a few specific features:

• Access to system-reserved features
• Ability to create system services
• They cannot be uninstalled

System applications cannot be removed in any way, as some of them are critical for the system to function properly.
Native applications which are not system applications can be removed, though.

Services

Application can provide a service by specifying it in their manifest. The service will be run at startup with the usual application's permissions.

When an application uses a service, there is exactly one service process running per active user.

Libraries

Libraries allow to share a program between multiple applications.

• How libraries work
• Dependencies management and resolving
• System libraries

How libraries work

Unlike applications, libraries can be installed in multiple versions. This means you can have three versions of the same library installed at the same time on your computer.

When an application wants to use a library, it explicitly indicates the list of versions it is compatible with. The system then gets the related version and provides it to the application.

A library by itself cannot do itself: no background process, no commands exposure, no permission granting. Applications simply use libraries as helpers to achieve specific tasks.

For instance, the system library fs which is natively available allows to manipulate files and directories easily.

Dependencies management and resolving

Each application indicates in its manifest the list of libraries it requires.

When the application is installed, the system will also check if the required libaries are already installed, and with the matching versions. If this is not the case, the library will be downloaded and installed as well (even if it's a volatile application).

When an application is removed, the system looks for each of its dependencies. If the dependency is not used by any application anymore, it is removed by default.

System libraries

The system provides several system libraries, which work are libraries that communicate with the system through signals to enable system-related and hardware-related features. Note that's it's possible to use signals directly to deal with the system and the hardware, but it's a lot more complicated than just using these system libaries.

• fs : Filesystem management
• net : Network communications
• ipm : Inter-process management (create processes, workers, IPC, shared memory, ...)
• gui : Graphical user interface library (relies on desktop)
• apps : Applications management
• perm : Permissions controller
• shell : Shell interface (run commands, ...)
• input : Input interface (keyboard, mouse, microphone, ...)
• sound : Sound interface
• system : System interface (control panel, low-level changes, updates, ...)
• sandbox : Sandboxes management (run applications in sandboxes, ...)
• desktop : Desktop management (desktop, windows, notifications, ...)
• hardware : Hardware management (drivers and devices)
• reactive : Reactive (relies on reactive, includes Reactive Markup Language, ...)
• sysver : Exposes the system version, its main purpose to indicate which system version is required for an application

Users

• The concept
• Users type
• Dangers of an admin. account
• User Privileges Elevation (UPE)
• Complexity level
• Users' data encryption
• Child and supervised users
• Groups
• User privileges

The concept

Many computers are intended to be shared by multiple persons. In such case, it is useful to separate the data of each user and to prevent other users from accesing another's.

By default, there is two user accounts: the System and the Administrator. They are called virtual users because it's not possible to log in these accounts. Other users (the ones which are created manually) are called custom users.

Each custom user has a dedicated data directory called the homedir, in /home/[username], as well as a list of files and directories it can read and/or write. By default, each user gets access to:

• Its homedir in /home/[username] ;
• The mounted periphericals in /media ;
• Its temporary directory in /tmp/[username]

Users type

Each user is of a specific type:

• Phantom: the user's data are erased after the computer is turned off ;
• Standard: nothing special
• Administrator: can run programs directly as administrator
• Main administrator: administrator that can manage storage encryption

When a user wants to perform a task it does not have the privileges to, it can (by default) ask to run the task as another user. The other user's credentials are then required.

It's not possible to ask to run a task as system or as administrator as these accounts are virtual and do not have passwords as such, except if an administrator user is tries to run a task as administrator.

Note that there is always one and exactly one main administrator. This user account is created during the installation process, and can be transferred later on to another administrator account.

Dangers of an admin. account

The problem with administrators account is that they can do almost anything (except some very specific things like editing the system's files). They can change system settings, read and change other users' data, and even run background processes at startup. They also have all possible privileges as they can edit their own.

This is why it's extremely discouraged to have two administrator users on the same computer, unless the two accounts are used by really trustworthy persons. As such, a large warning is shown if you try to create a new administrator user.

User Privileges Elevation (UPE)

Users can ask to perform a task with the privileges of another user, such as running a program as administrator. This uses the User Privileges Elevation (UPE) system, builtin the sys::perm service.

In such case, the program is still run as the current user, but with the privileges of both the running user and the user specified to the UPE.

Running a program with UPE requires to know either the other account's password, or to have an authorization from this user. For instance, admin. users have an authorization to use the Administrator account, without providing any password, although a human confirmation is required.

Each request, successful or not, is logged in the log file at /etc/logs/auc.

Complexity level

Each user can define a complexity level, which will affect its default settings.

A higher level of complexity will make the system display more informations, give more details about errors that may require some technical knowledge but providing additional features and informations in exchange.

The levels are:

• Beginner: only show basic informations, make the permission popups as simple as possible
• Standard: the default complexity level
• Advanced: shows additional informations, displays some error reports, makes the permission popups more precise and verbose
• Power: shows very detailed informations, displays all error reports, makes the permission popups display exhaustive informations

Users' data encryption

See encryption.

Child and supervised users

Child users are users that are supervised by the parental control. Supervised users are users that are part of a domain.

Groups

Groups are a set of privileges defined by the administrator. Access to some folders or application's permissions (e.g. microphone access) can be restricted to a specific group, for instance. When a user is created, it can be put in a group. It then automatically inherits all of the group's privileges, in addition to his own ones. The administrator can add and remove users from groups with the control panel.

When a user is in a group, the group's privileges cannot be revoked for this user.

User privileges

User privileges indicate the list of actions a user can perform or not. You can find more in the specifications of the sys::perm system service.

Features

This folder contains explanations about each major feature of NightOS. Each document is designed to be relatively easy to read and understand.

• The balancer - improve performances by balancing processes' priority
• Crash saves - prevent data loss at maximum with crash-proof data saving
• Domains - manage a network of computers
• Encryption - encrypt the whole storage and individual user accounts
• Freeze-prevention system - prevent the system from freezing when all RAM and CPU power are used
• Parental control - manage children access to the computer
• Permissions system - prevent applications and users from doing whatever they want
• Sandboxes - isolate applications to prevent them from harming important data
• Synchronization - synchronize settings between multiple computers

Balancer

The balancer is a kernel feature that allows to get more performances out of most important applications.

• How the balancer works
• Application processes suspension
• Performance mode

How the balancer works

The balancer allows to manage the priority of userland processes - and only them. Here is the list of its features:

• Increase priority of this application: gives a priority of 8 to the process linked to the active window ;
• Give maximum priority to this application: gives a priority of 10 to the process linked to the active window ;
• Set this application with maximum priority: always give a priority of 8 to this application's processes ;
• Suspend/resume this application: see below ;
• Enter performance mode: see below

Application processes suspension

Application processes can be suspended, which is an equivalent of pause where they don't run at all. This is achieved by setting their priority to 0.

A suspended application can then be resumed, and because it was just suspended it will instantly run again, without any data loss.

When a process is suspended, all its child processes are, too.

Performance mode

The performance mode performs the following actions:

• For all userland processes with a non-null priority, set their priority to 1 ;
• For all the processes of the application related to the active window, set a priority of 10.

This makes all other applications running a lot slower, but the current one will run a lot faster. The priority is re-calculated whenever the active window changes.

When a fullscreen application uses more than 50% of CPU in fullscreen, or when it asks for it, an overlay suggesting to enter performance mode is shown.

Performance mode is automatically exited when the related process exits.

Crash saves

Crash saves prevent most data loss caused by a crash, in all applications supporting it.

• How crash saves work
• Crashes detection
  • Invalid shutdown indicator
  • Application's crash indicator
• Restoration process

How crash saves work

Every minute (this delay can be changed in the registry), a SYS_CRASHSAVE_COLLECT answerable signal is sent to all running applications. Applications can then answer with their state data, which should contain every data required for the application in order to be restored to its exact current state later. They may join a title message, which is expected to be their main window's title - if they have ones. They can also answer with a specific message telling they won't give a crash save data during the current collect. The signal will still be sent on the next collect process (e.g. a minute later). A last answer method is with another message telling they won't give any crash save for the running instance. The signal won't be sent again for this instance of the application. This message is most of the time encountered when the application doesn't implement the crash save process.

If the application didn't answer when the next collect occurs, the signal is aborted, but the next one will be sent.

When a crash save has been collected for a given application, it is stored in /home/[user]/appdata/[appname]/crashsaves/[timestamp]_[pid].csf.

NOTE: Crash saves' intregrity is controlled using the system's integrity checker.

Crashes detection

Invalid shutdown indicator

When the system starts up, it creates an empty file in /etc/sys/awake. When the system shutdowns gracefully, this file is removed.

If, during startup, this file already exists, the system hasn't shutdowned gracefully. When this happens, a dialog message is shown, suggesting to re-open crashed applications with their last crash save.

For each application that do not have an available crash save (e.g. when the system crashed before a crash save could have been collected for this application), an indicator in the dialog box will show this application cannot be restored.

Application's crash indicator

When an application starts, the system creates an empty file in /home/[user]/appdata/[appname]/crashsaves/awake for user applications (even for global applications).

When the application exits gracefully, the file is removed automatically. When the system detects the application exited, if this file still exists, it shows a dialog box asking if the user wants to relaunch the application with the last crash save.

NOTE: If there is no available crash save (e.g. when the application crashed before a crash save could have been made), the dialog box will simply show the application crashed.

NOTE: Because a crash save could have been collected for several instances of an application, they can all be restored afterwise.

Restoration process

When a crash save is attempted to be restored, the sys::crashsave service. When the application is ready, a SYS_CRASHSAVE_RESTORE confirmable signal is sent, with the application's crash save. The application is expected to confirm the signal when it has finished restoring its state using the crash save. The crash save is not deleted directly, though. It is renamed using the new instance's PID and kept until the next collect process receives a new crash save for this instance.

If the application crashes before another crash save can be made and didn't confirm the restoration signal, when this process happens again, system will indicate the application appear to have crashed during crash save restoration.

Domains

Domains are a set of computers running NightOS which are linked to each other. The way a domain's computers behave and interact are defined in the Domain Supervisor, which is only available to the domain administrator user.

It behaves like an extension of the parental control, though they are two completely distinct features and domains offer a lot more features, while being made not to manage a child's access to a computer but to manage thousands of comupters at once.

• The concept
• Domain supervisor

The concept

Domains enable all features of parental control (without the dedicated application though), as well as the following ones:

• Mount a common storage between computers
• Use remote user accounts for log in
• Restrict access to applications
• Restrict available permissions to users
• Restrict installation and update of new applications
• Manage how applications and the system are updated
• Monitor CPU, RAM and storage usage on all computers
• Get access to every user's storage (unless per-user encryption has been explicitly allowed)
• Get remote terminal access to every running computer
• Get virtual desktop access to every running computer
• Put computers to sleep, hibernation, log out current user, power them off or reboot them
• Start any computer remotely (if the computer does support it, e.g. through PoE)
• Limit disk usage per user (in disk usage percentage or absolute value)
• Limit the session duration per user
• Limit the number of physical and virtual cores per user
• Limit the amount of memory per user

Domain supervisor

The Domain Supervisor is a system application that shows up on any computer that is part of a domain. Only domain supervisor users can see it, but every user can run it through command line (though it will ask for a domain supervisor's username and password).

Here is the list of options the domain supervisor proposes:

TODO

Encryption

Encryption allows to prevent unauthorized access to a data with a password.

• Global encryption
• Per-user encryption
• Combining global and per-user encryptions

Global encryption

By default, the whole storage can be encrypted using the computer's master password, which is prompted on startup.

During installation, even if this feature is turned off by the user, a random encryption key is generated in the bootloader, which can be accessed through the /etc/sys/gbpwd file. This key will be used to encrypt the storage when global encryption is active. This process allows administrators to change the master password whenever they want to.

When global encryption is turned on, the administrator must choose an encryption password it will communicate to each user. The system will not be able to start without this password.

On startup, the system will run a bootloader which is not encrypted (and thus vulnerable to attacks) which asks for the master password. This master password is then used to decrypt the master key in the bootloader. Then, the /sys/valid file is decrypted, and its content is compared to ValidMasterKey. If contents are equal, the provided master password is valid and the boot process starts normally. Else, the key is not the good one and the user is prompted a new one.

The point of global encryption is that external persons cannot read the storage's data by just reading the storage ; they must either know the master key or put a malicious bootloader to spy on the user.

NOTE: The ability to encrypt the storage globally, change the master password or decrypt the storage is by default reserved to the main administrator, but this privilege can be given to normal administrators, though this is highly discouraged.

Per-user encryption

But any non-guest user can also use a built-in system tool to encrypt its data using its own password. This way, the user's data become unreadable without his password, making even administrators unable to read his data.

The encryption/decryption key is generated automatically when the user account is created and stored in the user's profile data (in /etc/sys/users).

When enabled, the encrypted directories are:

• The homedir at /home/[username]
• The tempdir at /tmp/[username]

This feature is enabled by default, but can be disabled by the administrator. Also, it is disabled by default on domains.

Combining global and per-user encryptions

When both a user enabled encryption for its user account and global encryption is active too, instead of having to encrypt and decrypt the data twice the user's data will be encrypted using a key derived from the master key as well as the user's encryption key.

This allows to save a lot of time but still prevent unauthorized access if the user's password is weak.

Freeze prevention

Freeze prevention prevents nearly all computer freezes, by reserving a little amount of resources to the system. It is enabled by default, but it can be disabled during installation process or from the control panel.

• How freeze prevention works

How freeze prevention works

The system forbids applications to access a fixed amount of RAM, called the freeze prevention resource (FPR), which is by default the smallest between 1% of the total RAM and 8 megabytes.

When more than 99% of CPU and/or RAM is used, the system puts itself in freeze prevention mode (FPM), which consists in listening to a specific keyboard shortcut (by default, Ctrl+Alt+P), controllable using the keyboard or the mouse (with a special block cursor, not changeable by applications) which asks user if they want to terminate gently the more resources-consuming application instance (by sending the TERMINATE signal), to kill it, to kill all instances of the application, or to see the list of running application instances with how much resources they consume, to make a choice on another one.

If developer mode is enabled, another option is added to access a tiny shell containing a set of commands made to list and manage running applications in a more powerful way.

Parental Control

Parental control allows to manage a child's access to a computer running NightOS.

• How it works
• Integration
• Features
  • Restrict usage hours
  • Restrict session duration
  • Restrict access to applications
  • Restrict access to websites
  • Restrict mature contents
  • Restrict installation of applications
  • Controlling session remotely

How it works

Parental control works using (at least) two accounts: the parent's account, and the child's user account, which is related to as a child user. Note that multiple child users can be managed using parental control.

The parent account then gets access to an application called "Parental Control", which allows to access all parental control-related features for each child user.

It is also possible to get important notifications by email, and even to install a smartphone application (iOS/Android) which brings the parental control application to phones, allowing to manage child users even when not at home.

Integration

Parental control's settings are available to third-party applications using specific permissions. This way, some applications can provide an additionally layer of security for child users, like preventing access to adult contents in a game store.

Features

Restrict usage hours

This feature allows to restrict usage of the computer by the child user during specific hours in the day. These hours can be set independently for each day of the week, allowing for instance to set specific hours in the week-end. It's also possible to set specific hours for some specific weeks or periods of times (e.g. for four specific days in a row), allowing for instance to set specific hours for vacations.

Child user will not be able to log in outside of these hours. If they are logged in when the upper limit is reached (e.g. only 8 AM to 9 AM is allowed and the child user is still logged in as 9 AM), a warning message will be shown 15 minutes before reaching the time limit, 5 minutes before, and 1 minute before, allowing the child to save its data.

When this final minute comes, a pop-up shows telling the session is going to be closed in 60 seconds, and a cooldown is shown in a closable pop-up balloon in the taskbar. An option is also shown to extend the session by up to fifteen minutes, sending a notification to the parent's phone as well as an e-mail.

The parent can then do nothing and the session will extend for a maximum of fifteen minutes, or reject it and the computer will shut down after 1 minute. Then another option shows to ask if the parent wants to permanently disable time extension.

Session time extension can be enabled/disabled during the parental control installation process (enabled by default). If disabled, the time extension option will not be shown to the child user during the timeout.

Restrict session duration

This feature allows to restrict how much time a session can be active, allowing for instance to limit a child user's usage to one hour per day. Like the usage hours, it can be set independently for each day of the week, for specific weeks, and for specific periods.

Session time extension applies here as well, and works the same way as for restricted usage hours. The related settings are independent though, allowing for instance to enable session time extension for usage hours but not for session duration, or the opposite.

Restrict access to applications

This feature restricts access to some applications, either using a whitelist (listing all the allowed applications), or using a blacklist (listing all the forbidden applications).

It's possible to automatically while/black-list new applications.

Restrict access to websites

This feature restricts access to some websites, either using a whitelist or a blacklist. It's also possible to apply these restrictions only to compatible web browsers. Whitelist is strongly discouraged outside of web browsers-only usage, as it may prevent some applications to work properly.

Whenever an application tries to access a forbidden website, the request will fail with a specific code indicating it's forbidden by parental control. Applications can still enforce this rule but this requires a specific permission that cannot be allowed by child users.

These restrictions are enabled through the built-in firewall, Vortex.

Restrict mature contents

The child user's birth date can be declared during setup, so its age (not its birth date) will be available to every application which asks for it (with a specific permission, which is by default automatically granted to all applications which ask it).

This allows, for instance, for web browsers to block websites that declare themselves as showing adult contents. Or even more precise protection, like a movies store not showing movies whom age limit is beyond the child user's age.

Restrict installation of applications

This feature prevents child users from installing and running volatile applications. By default, installation is allowed but sends a notification to the parent user asking if the child user is allowed to install the requested application ; while running volatile applications is simply disabled.

Controlling session remotely

This feature allows parents to log out the child user remotely. This will show a pop-up indicating the session will be closed in 60 seconds, behaving just like usage hours restriction. It may also provide session time extension based on how the log out was triggered (log out the child user or log out the child user and disable session time extension) and the setting selected during the setup process.

Sandboxes

Sandboxes allow to test an application without applying modifications to the system.

• How sandboxes work
• Persistent sandboxes
• Puppet sandbox

How sandboxes work

A sandbox is an execution mode where an application's modifications to the disk are not applied directly to it, but instead to a virtual drive stored in its sandboxes folder. When the app. exits, a confirmation overlay proposes to apply the modifications to the real storage - it's also possible to see the changes before applying them.

In developer mode, it is possible to export sandboxes and import them on other computers.

Persistent sandboxes

Sandboxes can also be created as persistent, which prevents them from being removed after the app. exited. Instead, the next time it will run, the same sandbox will be used again.

Puppet sandbox

Sandboxes can be controlled by another application (this requires an administrator permission, though). This allows to automatically accept or decline every API usage, like permissions or I/O requests. Such sandboxes are called puppet sandboxes, and they can be especially useful for testing.

Synchronization

Online backup and synchronization is performed through Cloudy.

Technical

This folder contains technical documents which essential describe low-level parts of NightOS.

• The controller - permissions management system
• Developer mode - enable powerful development options
• File formats - description of all native file formats
• Filesystem Abstraction Layer - how the system ensures specific features on less powerful filesystems
• Integrity checker - ensure the system hasn't been corrupted
• I/O manager - manage input/output requests
• Inter-process communication - communication between processes
• Multi-platform management - how the NightOS ecosystem can be used on other operating systems
• Performances - system tweaks used to optimize general and specific-case performances
• Pre-compiling applications - pre-compiling applications to improve installation time and size
• Processes - low-level view of how code runs in a concurrent way
• The registry - configure the system's behaviour and features
• Services - special processes that run in the background and allow other applications to perform specific tasks
• The shell - the de-facto way to run complex and/or automatized tasks on NightOS

Controller

The controller is a system library that manages permissions of processes. The I/O manager relies on it to accept or reject requests.

• Notion of scope
• The perm system library
• Permissions extension

Notion of scope

Permissions are split into several scopes:

• The application scope contains the permissions a given process is borned from ;
• The user scope contains the permissions the user who launched the application has ;
• The mode scope contains the permissions the execution mode (either system or userland) has.

The mode scope restricts the user scope, which itself restricts the application scope. This means that, if the application scope specify a permission that is not covered by the user scope, it is not applyable to the process. This allows to prevent applications and users from getting too high permissions.

The mode scope prevents applications from performing harmful tasks such as writing the system. Only system applications, which run in system mode instead of userland mode, gets an unrestricted mode scope.

The perm system library

The perm system library is an interface for the controller which allows processes to check their own permissions, ensure they can make I/O requests before effectively making them, and extend their permissions (see below).

Permissions extension

A process can, at any moment, send a permission extension request (PER) using the perm library. It allows to gain a new permission by showing an overlay the user can accept or decline. If the permission is accepted, the requested permission is added to the process' one - and sometimes to the application's one.

If the requested permission is out of its maximum scope (e.g. asking for write access to /etc while being ran as standard user), the request is rejected.

Developer mode

The developer mode enables several features that are not available to default users. Only administrator users get access to them when they are enabled.

• How to enable
• Features
• Application proxies

How to enable

Developer mode can be enabled either by typing the following command in the terminal:

adm central --enable dev-mode

Or, while holding the Ctrl Alt and Maj key, type DEV (three letters).

A warning dialog will appear. If you confirm, it will display an UPE dialog for the administrator account. Then a final confirmation dialog will ensure you're sure about what you're doing. Then development mode will be enabled (no reboot is required).

Features

• Additional features will appear in the control center (see the options)
• The registry can be imported and exported

Application proxies

Application proxies are applications that intercept all applications calls to the system on-the-fly. It is useful for applications debugging.

When a proxy is set up for an application, all system calls sent to this application will be transferred to the proxy, which will be able to do whatever it wants with it. It's possible for the proxy to never answer the signal, to change its content before actually sending it to the system, etc.

Basic usage of an application proxy is as a "listener": it only listens to specific signals and logs them, without cancelling them and/or modifying them. This is useful to check all filesystem access requested by an application, in real time.

Devices

• Connecting a new device
• Interacting with a device
• Device handler files persistence
• Custom device handler filename

Connecting a new device

When a new device is connected to the computer, a device handler file (DHF) is created in the /dev directory (see filesystem structure).

Depending on its type, it will be put in a different directory, as shown in the above document. Note that new category directories may be introduced later on.

The filename itself is a random unique identifier made of a single lowercase letter, a digit, another lowercase letter and finally another digit.

For instance, if a hard drive is connected to the computer, the DHF may be something like /dev/sst/f5t2.

Interacting with a device

Device handler files are not simple files ; they can only be used through the sys::hw service.

Different actions may happen depending on the device's type:

• Camera devices: when an application asks to capture a photo/video, the device will be suggested to take the images from
• Microphones: when an application asks to capture sound, the device will be suggested to capture the sound from
• Sound output devices: the device will be available for playback
• Network adapters: the device will be available to make network requests on
• Other supported wireless devices: depends on the type (Bluetooth adapter, ...)
• Basic/persistent storage devices: when possible, the device will be automatically mounted in /mnt and visible in the files explorer

There are many different types of devices, all can be found in the specifications of the sys::hw service under the "Driven device type" (DDT) section.

For uncategorized devices (in /dev/etc), a popup is shown to the user, to indicate the connected device is not recognized. Some applications may still be able to interact with it (for instance, a storage device using an unsupported protocol).

Device handler files persistence

When a device is connected, its DHF is not removed. Instead, if a process tries to interact with the DHF, the sys::hw service will indicate the device is currently not connected.

When the device is connected again, it is associated to the same DHF again. This allows applications to persist the device's location and use it later on.

Custom device handler filename

It's possible to give a custom handler filename for intuitivity. In such case, a new DHF is created in the related directory, but the old DHF is kept anyway. This allows to not break compatibility with applications that rely on the old DHF.

The two DHF will point to the exact same device, without any difference. The custom DHF can be removed anytime, although this is discouraged as applications may rely on it.

Also, custom DHF names must always be longer than 4 characters, to avoid confusion with automatically-generated DHFs.

File Formats

This document presents the file formats natively handled by NightOS. Some files are only handled by optional first-party applications that may not be installed on the computer, so some formats may not be supported in specific installs where such applications have been removed/were never installed.

• Common formats
• Virtual storages
• Application packages
• System updates

Common formats

Common file formats are natively handled:

• Text files (.txt, .md, ...) with Gravity
• Audio files (.mp3, .flac, ...) with Sonata
• Image files (.png, .jpg, ...) with ShootingStar
• Video files (.mp4, .mkv, ...) with Milkshake
• Archive files (.zip, .tar, ...) with Blackhole
• E-book files (.cbz, .cbr, ...) with Reader
• E-mail files (.eml, .vcf, ...) with Postal
• Web files (.html, ...) with Rocket
• Virtual storage files (.iso, .vfd, ...) with Locky

Virtual storages

Virtual storages are files that contain one or several virtual filesystems. Different filesystems can be put in, but all must be supported natively by NightOS in order to properly work without needing to install any additional application.

The filesystems can be encrypted individually. The whole storage can also be encrypted.

They have multiple purposes: store encrypted files, virtual filesystems for sandboxed applications, etc.

Virtual storage files have the .vts extension and are opened using Locky.

Application packages

Applications can be installed from standalone files called application packages. Installable applications have the .nap (NightOS Application Package) extension, while volatile applications have the .nva (NightOS Volatile Application) one.

They are opened using Skyer.

System updates

The system is intended to be updated through the update section in the settings, but sometimes it may be required to install updates offline for specific reasons (e.g. no network connection available at the moment). In such case, it is possible to download incremental updates as system update files.

They may contain one or several updates, and are only installable on a very specific version of the system, to avoid missing some other incremental updates.

System update files have the .nsu (NightOS System Update) extension and are opened using the control center.

Filesystem Abstraction Layer

A filesystem is the way to deal with a storage device, for instance to store files and directories.

NightOS supports almost all filesystems, but none support all of the required features, which is why a filesystem abstraction layer (FAL) is added to ensure the filesystem supports specific features. It's basically an interface that runs when dealing with the internal storage, that ensures its capabilities.

The layer works by storing the informations in a small file dedicated to this purpose.

• Structure
• Symbolic links
  • Concept
  • Cyclic symlinks
• Flows
  • Concept
  • Creating a flow
  • Connecting to a flow

Structure

The filesystem structure of a NightOS installation can be found in the related specifications document.

Symbolic links

Symbolic links, abbreviated symlinks, are files that point to another location.

Concept

A symlink points to a specific item: file, folder, device, anything. It's just not a shortcut, though, as the symlink will still work if its target is moved.

When a symlink is accessed, the system will transparently access its target item instead.

When a symlink is removed, it does not affect the original target. Also, any number of symlinks can target the same item, and symlinks can target other symlinks to. When accessing a symlink, if its target item is a symlink itself, the latter's target will be accessed instead, and so on, until we do not encounter a symlink anymore.

This can be explicitly disabled when interacting with the filesystem, or limited to a specific number of children.

Also, symbolic links may point to a location on another storage.

Cyclic symlinks

Given the following situation:

1. We create a symlink A which points to a random file
2. We create a symlink B which points to A
3. We update the target of A to be B

When we will try to access A, the system will access B, then A, then B, and so on. This is called a cyclic symlink chain. In such case, the chain is reduced to the minimum (for instance, if we had C pointing to A, the minimum chain would not be C A B but just A B), and marked as erroneous. The process that tried to access the symlink will receive a specific error code to indicate a cyclic symlink chain was encountered.

Flows

Flows are a simple and efficient way for processes (mostly services) to allow treating flows of data.

Concept

A flow is a file without extension, located in the /fl directory, that can either send data to reader processes (read-only) or receive data from writer processes (write-only).

To understand the concept better, here is the list of native flow files that are always available:

Processes are based on pipes.

Creating a flow

When a process wants to create a flow, it follows the following procedure:

1. The process asks the sys::flow service to create a flow
2. The service creates the related flow file in /fl
3. When a process reads from the (readable) flow file, all data is continuely retrieved from the creator's SC (until the flow is closed)
4. When a process writes to the (writable) flow file, all data is continuely written to the creator's RC (the flow is not closed after that though)
5. When the creator closes its SC/RC, the IPC channels duo is closed and the flow file is removed

Connecting to a flow

When a process wants to read from or write to a file, it first asks the sys::flow service to connect to this file. If accepted, it receives a SC or RC to interact with the flow.

Integrity checker

• System immutability
• Hashes computing
• Checking hashes at startup

System immutability

The /sys directory is immutable, which means it cannot be modified, even by the main administrator. The only process where this directory may be affected is during a repair process or during an update, which are both processes that are handled by the system directly.

Hashes computing

When critical files are written, in /sys or /etc/sys, their hash is computed using the SHA-384 algorithm and stored in the system's hash registry (/etc/sys/hashes file).

Checking hashes at startup

When any of these files is read from the disk, their content is computed and compared to the hash registry. If the hashes don't match, the file is considered as corrupted. The consequences depend on the location of the file:

• System (/sys): the system will check its backup integrity and then restore itself from the backup ;
• System backup (/sys/backup): the system will check /sys integrity and then make a new backup of itself ;
• System + system backup: the system won't boot and indicate it must be reinstalled (documents won't be lost if they aren't corrupted) It's possible to force the boot process, but a large warning will indicate it may cause crashes or even introduce security issues ;
• Hash registry (/etc/sys/hashes): same thing ;
• Registry (/etc/sys/registry): same thing ;
• Crash save: the crash save won't be restored (can be forced but with a large warning) ;
• Sandbox: system will refuse to start the sandbox (can be forced but with a large warning) ;

IMPORTANT: If some malicious person edits the system files and its backup, it can also edit the hash registry, so the error won't show up. So that's not because there is no error the system is safe and has not been modified!

I/O nano-manager

The Input/Output Nano-manager, formerly known as Ion, is a part of the system which treats input/output requests from processes.

• Hardware access
• Requests
• Permissions
• Requests priority

Hardware access

When a process tries to access the hardware, it must go through Ion, which will allow it or not to interact with the desired component.

System services such as sys::fs or sys::net use Ion to deal with the related hardware components.

Requests

When a process tries to access, for instance, the filesystem, it sends an I/O request to the manager which is pushed in an internal queue. The manager sorts incoming requests and treat them depending on their arrival.

Permissions

To ensure a process has the required permissions to make a specific I/O request, the manager asks the controller to verify its permissions. The controller's response makes the manager accept or reject the request.

Requests priority

Requests are treated by priority, which is made both of its arrival timestamp (first one, first out) but also of the process' priority: a process with an higher priority will see its I/O requests treated more quickly.

Multi-platform

NightOS programs can be ran either on Linux, Windows or MacOS systems using a wrapper called the NightOS Calls Wrapper (NCW). It can be downloaded from NightOS' official website.

When installed on a system, it allows to use NightOS applications natively, although permissions management will not be as smooth and many options will not be available (e.g. no desktop, no access to native NightOS applications - only native libraries, etc.).

Performances

• Cycle and phases
• Processes' priority
• Balancing performances

Cycle and phases

Because processes' instructions cannot always be processed parallely, the system will let each application run a fixed amount of instructions, before going to the next one. This short period of time is called a phase and generally lasts a few milliseconds. The treatment of a single phase for all running processes is called a run cycle. When a process isn't in a phase, it is "paused" until the next one.

Processes' priority

Each process is started with a given priority, which is a number from 0 to 10, which can be modified during its execution. The more priority it has, the more instructions it will run in a single phase.

A priority of 0 prevents the program from running any instruction at all. In this case, only an external process will be able to increase the priority to let the program run again.

By default, a background process gets a priority of 2, a visible process gets a priority of 4, the process linked to the active window gets a priority of 6, and the process linked to a fullscreen window gets a priority of 7.

When a process asks to set its priority to 9 or 10, a confirmation overlay is shown.

Balancing performances

See the performances balancer.

Pre-compiling

On traditional operating systems, programs are provided as binary and so are only available for one specific platform (Windows, Linux, MacOS, ...) with a specific architecture (x86, ARM, ...).

In order to prevent this, and in order to bring more stability and security to native programs, NightOS programs are shipped as NightOS Pre-compiled Programs (NPP) using a specific language called CommonAssembly.

• How it works
• Compatibility
• Usage

How it works

CommonAssembly is very similar to WebAssembly: compressed very low-level source-files that are built ahead of time on the machine that will run it. This enables several advantages:

• Programs are multi-platforms, multi-architectures
• Programs run faster thanks to being optimized for the specific machine they will run on
• Programs are still very fast to compile
• Source code is protected for closed-source applications as CommonAssembly is made of basic instructions
• Better security thanks to programs being checked ahead-of-time

Compatibility

CommonAssembly can be ran on other operating systems than NightOS using a lightweight wrapper available on NightOS' official website.

NOTE: While CommonAssembly is multi-platform, NightOS applications take advantage of NightOS' features like more powerful signals that are not natively available on other systems. In order to run such applications, you must install the full wrapper which is available on the same website.

Usage

Open-source and closed-source NightOS applications usually bring pre-compiled versions of themselves in order to fasten a lot the installation process, and to get rid of the need of having the heavy build toolchain installed on the target computer.

This allows to install applications really fast, and to optimize them for the current machine.

Processes

WARNING: This document is far from being complete and may lack a good description of what processes are, and/or the features they offer.

NightOS processes are implemented a bit like Linux' ones, with additional features. There are several types of processes:

• System processes, which are created by the system ;
• Application processes, in which applications run ;
• Worker processes, in which applications' workers run

The base and system processes are called low-level processes, while application and worker ones are called userland processes.

You can find the implementation details of processes in the kernel document.

• User privileges
• Child processes
• Threading
  • Main thread
  • Thread-local storage
• Automatic permissions inheritance

User privileges

Each process is ran as a specific user, which determines the maximum allowed scope for controller requests, and with a list of initial privileges (the ones given to the application).

Child processes

A process can create child processes (it's called a fork). The child process will roughly be a 1:1 copy of the parent process, but with its own unique PID.

Child processes automatically inherit their parent's permissions.

When a process exits, all its child processes are immediatly killed. It's up to the process to ensure its children are properly terminated before it.

Threading

A process can create threads, which are still a part of the process. Threads allow to run multiple part of a process concurrently, as the kernel may run several threads in different processor cores.

All threads share the same address space and memory, although they all have a reserved space called the thread-local storage.

Threads work as a hierarchy ; when a thread creates another, it is called the new thread's parent, while the new thread is its child. When a thread terminates, all its children are instantly destroyed.

Main thread

When a process starts, its instruction run its main thread. Due to threads being hierarchised, exiting the main thread will result in all other threads being closed immediatly, which is why the main thread should always first terminate its children properly to ensure all data are synchronized for instance.

Thread-local storage

Threads have a reserved portion of memory in their address space called the thread-local storage (TLS).

Automatic permissions inheritance

When an application process gets a new permission, all other processes from the same application inherit it, unless this permission is granted only for this instance of the application.

Registry

The registry contains all system configuration.

• Format and content
• Debugging

Format and content

The registry's format and content can be found in the specifications document.

Debugging

In developer mode, the registry can be exported to the HFRR format and imported back.

Services

A service is a process that launches at startup and may receive IPC messages from any process.

• System services
• Application services

System services

System services are native applications. They have system permissions and as such are allowed to perform any task. Their purpose is to allow processes to perform specific actions like permissions management or filesystem access without hunging up the kernel.

Each system service is a system application, exposing a service. Most also expose command-line tools.

You can get more details about how services work in the specifications document.

You can find the list of system services in their specifications directory.

Application services

Application services, also called userland services, are provided by applications themselves.

Shell

The shell, called Hydre, is the part that interprets scripts.

• Commands
• Pipes
• Specifications

Commands

Hydre works using commands, which can take arguments. When running a command which comes from an applications which expose it, the said application is run in a new process which uses its execution context.

The special thing about running an application from one of its exposed commands is that its CMDIN, CMDUSR, CMDMSG, CMDERR, CMDOUT and CMDRAW pipes are exposed to the caller.

You can find more informations about how a process can interact with a running command in the IPC documentation.

Pipes

When a command is run from Pluton, the command process' pipes are handled as follows:

• All user inputs are sent to the CMDUSR pipe
• All messages sent through CMDMSG and CMDERR pipes are printed as they are received by the shell
• The command's return value (from CMDOUT) is printed after the command completes
• Data written to CMDRAW is not shown, as they don't have to be UTF-8 encoded, or even to be a string. They can be redirected through pipes, though

Specifications

Hydre's specifications can be found in the related document.

Specifications

This folder contains specification documents, whose purpose is to describe fully a specific concept or feature. Think of these as reference documents - they are not meant to be easily understood without a solid knowledge of how NightOS works. For more informations about low-level concepts these documents refer to, you can check the technical documents first.

• Application context - launch an application to directly perform a specific task
• Applications package - files representing a whole application
• Applications manifest - how applications describe themselves in their package
• Filesystem structure - list of file and directories and their meaning
• Inter-process communication - communication between processes
• Libraries - what are libraries
• The registry - exhaustive specification of the registry's content
• Vocabulary - the list of NightOS-related terms
• The shell - how Hydre works
• Shell scripting - Hydre's scripting language
• Signals - complete specification of signals
• System calls - complete specification of system calls
• Kernel - complete specifications of the kernel
• System services - complete list of system services

Execution Context

An application's execution context is a piece of data that is provided when the application starts.
It indicates why the application was started and so what it is supposed to do.

• Startup Reason
• Context header
• Arguments structure

Startup Reason

The most important information is the startup reason, which indicates why the application was started.

It is one-byte long, and is made of the following bits (starting from the strongest):

• Bits 0-3:
  • 0x1: the application was started as part of its post-installation process
  • 0x2: the application was started as part of its pre-update process
  • 0x3: the application was started as part of its post-update process
  • 0x4: the application was started as part of its pre-uninstallation process
  • 0x4: the application was started by the system as an application service
  • 0x5: the application was started by the desktop environment
  • 0x6: the application was started by itself (from another process of the same application)
  • 0x7: the application was started by another application
  • 0x8: the application was started using one its exposed shell commands
• Bit 4: set if the application was started automatically after a crash/improver shutdown and should to the sys::crashsave service to get a crashsave
• Bit 5: set if the application's raw output (CMDRAW) will be read (e.g. through the use of a shell operator)

The startup reason is especially important as it determines what the application should do (e.g. uninstall itself, run as a command...) but also if it should output data through its CMDRAW in case it was called by a command.

Context header

The context header is stored as a single block of data, consisting of:

• The startup reason (1 byte)
• Ambiant informations (1 byte)
  • Bit 0: set if the application is starting for the very first time since it was installed
  • Bit 1: set if the application is starting for the very first time for this specific user
  • Bit 2: set if the application is starting for the first time after an update
  • Bit 3: set if other instances of this application are running
• The application's ANID (4 bytes)

If the command was not started a command, the context ends here. Else, it also contains the following informations:

• The ANID of the caller application (4 bytes)
• The number of arguments the process was started with (1 byte)
• The cumulated size of all arguments, in bytes - up to 63.5 KB (2 bytes)
• RC identifier for the CMDIN pipe (8 bytes)
• RC identifier for the CMDUSR pipe (8 bytes)
• SC identifier for the CMDMSG pipe (8 bytes)
• SC identifier for the CMDERR pipe (8 bytes)
• SC identifier for the CMDRAW pipe (8 bytes)
• SC identifier for the CMDOUT pipe (8 bytes)
• Future-proof shift space (196 bytes)

Arguments structure

The context header is followed by the list of each command-line argument, taking up to 64 KB.

Arguments are a simple concatenation of encoded values.

Application packages

Application packages are files that have either the .nap (NightOS Application Package) or .nva (NightOS Volatile Application).

• Content
• Manifest
• Pre-compiled applications

Content

NAP and NVA files are ZStandard archives which only requirement is to contain, at the archive's root, a manifest.toml file describing the archive itself, a hash.md5 ensuring the archive has not been corrupted.

Manifest

The manifest format can be found in the related specifications document.

Pre-compiled applications

By default, and if possible, the system will always try to install pre-compiled programs from applications' package. If the pre-compiled programs are not available, it will be built from source code - which takes a lot more time.

Application Manifest

Here is an example of an application manifest (REQ: required, OPT: optional):

# [REQ] Informations about the application
infos:
  # [REQ] Application's name, as shown when installing it
  name: Cloud Notepad App
  # [REQ] Application's slug, as stored on the disk in the /apps directory
  slug: cloud-notepad-app
  # [REQ] Application's description, as shown when installing it
  description: A notepad application that allows syncing your files in the cloud
  # [REQ] Application's version, following semantic versioning
  version: 1.0.0
  # [REQ] Application's authors
  authors:
    - name: Clément Nerma
      email: clement.nerma@gmail.com
  # [REQ] Application's icons (in the archive)
  # [REQ] "%{}": either 16, 32, 64, 128 or 256 pixels (icons must be square)
  icon: assets/icons/app/%{}.png
  # [REQ] Application's license (must be in a list of available licenses)
  license: Apache-2.0

# [REQ] Application package's content
content:
  # Packages can either contain source code only, pre-compiled programs only, or both
  # <for source packages> [REQ]
  source:
    # [REQ] Build tool (must in the list of the toolchain's supported build tools)
    toolchain: rust
    # [REQ] Required build tool-related options
    build: {}
    # [OPT] Build tool-related options
    options:
      optimize: O3

  # <for precomp packages> [REQ]
  precomp: main.npp

# [OPT] Event triggers
events:
  # [OPT] Should the application start just after being installed?
  postinstall: false
  # [OPT] Should the application start just before being uninstalled?
  preuninstall: false
  # [OPT] Should the application start just before being updated?
  preupdate: false
  # [OPT] Should the application start just after being updated?
  postupdate: false

# [OPT] Exposed commands (see the related document for additional informations)
commands: {}

# [OPT] Does the application expose a service?
service: false

# [OPT] Additional informations
additional:
  # [OPT] Available languages (in a list of existing languages)
  languages: ["en-US"]

# [REQ] Application's dependencies
dependencies:
  # [REQ] Required libaries
  libraries:
    sysver: ^1.0.0 # Any stable version

  # [OPT] Required fonts
  fonts:
    fonts:open-sans: true # Any version

For more informations about exposed commands, see the related document.

Filesystem hierarchy

This document presents the filesystem's hierarchy.

• Hierarchy
• Notes

Hierarchy

NOTE: <F> indicates the item is a file.

/
├── app                            Interactables available to all users
│   └── <appname>                  An application's folder (NOTE: one sub-folder per version for libraries)
│       ├── content                Application's program (executables, static resources, ...)
│       ├── crashsaves             Application's crash saves
│       ├── data                   Application's data (e.g. database)
│       ├── packages               Application's packages (original package + update packages)
│       └── sandboxes              Application's sandboxes
├── dev                            Connected devices
│   ├── cam                        Cameras
│   ├── bst                        Basic storage devices (SD cards, USB keys, ...)
│   ├── etc                        Uncategorized devices
│   ├── mic                        Microphones
│   ├── net                        Network adapters (Ethernet adapter, WiFi card, ...)
│   ├── snd                        Sound-related output devices (Sound card, DAC, ...)
│   ├── sst                        Sensitive storage devices (Hard drives, SSDs, ...)
│   └── wrl                        Other supported wireless devices (Bluetooth adapter, ...)
├── etc                            Mutable data folder
│   ├── env   <F>                  Environment variables
│   ├── fal   <F>                  Filesystem abstraction layer (1)
│   ├── hosts <F>                  Hosts overriding (e.g. 'localhost')
│   ├── lock                       Opened lock files
│   ├── logs                       Log files
│   |   └── upe                    History of UPE requests (2)
│   ├── public                     Public data, readable and writable by everyone
│   └── sys                        System's mutable data - available to system only
│       ├── registry               System's registry
│       ├── awake    <F>           System's shutdown indicator to detect if there was an error during last shutdown
│       ├── hashes   <F>           Critical files' hashes for the integrity checker (3)
│       ├── gbpwd    <F>           Global storage's encryption key (4)
│       └── users    <F>           User profiles and groups
├── fl                             Flow files
├── home                           Users' data
│   └── <user>                     A specific user's data
│       ├── apps                   User's applications (same structure as for `/apps`)
│       ├── appdata                User's applications persistent data (not removed when the application is uninstalled)
│       ├── desktop                User's files appearing on the desktop
│       ├── documents              User's documents
│       ├── downloads              User's downloads
│       ├── music                  User's music files
│       ├── pictures               User's pictures
│       ├── videos                 User's videos
│       └── trash                  User's trash
├── mnt                            Mounted storages
│   └── root                       Soft link to `/`
├── sys                            System - immutable outside of installation, repair processes and updates
│   ├── apps                       System applications
│   ├── boot                       System's boot program
│   ├── langs                      Translation files
│   ├── old                        Old versions of the system, used during the repair process (compressed archives)
│   ├── backup                     Copy of the last system version (compressed archive)
│   ├── kernel                     Custom micro-kernel
│   └── valid   <F>                A file that just contains "ValidMasterKey" to test if the provided master key is valid at startup
├── tmp                            Temporary folder (cleaned during shutdown)
│   └── <user>                     Temporary folder for a specific user

Links:

• (1) Informations used by the Filesystem Abstraction Layer
• (2) UPE requests
• (3) The integrity checker
• (4) Global storage's encryption key

Notes

Globally installed applications (located in /apps) can store user-specific data in /home/[user]/appdata/[appname], which will only be made of the data, crashsaves and sandboxes folder.

Inter-Process Communication

This document describes the way Inter-Process Communication (IPC) works.

• Pipes
  • Opening pipes
  • Pipes' pending data
  • Pipes locking
  • Closing pipes
  • Message pipes
  • Interactive usage
• Service sockets
  • Opening
  • Exchanges and messages
  • Concluding exchanges
  • Methods and notifications
  • Closing service sockets
• Shared Memory

Pipes

Inter-process Uni-directional Channels (IUC), also called pipes, allow two distinct processes to communicate.

Pipes are made of two uni-directional parts:

• A sender channel (SC) which can only be written to (write-only)
• A receiver channel (RC), which can only be read from (read-only)

The two processes sharing a pipes are:

• The sender process, which uses the SC to send data to the other process
• The receiver process, which uses the RC to retrieve data sent by the other process

The process that creates the pipe gets both the SC and the RC, and is expected to provide one of them to another process.

Each SC and RC has a unique identifier, which is binded to the process that created it.
The process receiving an SC or RC receives another identifier for it, unique to that process, which prevents IDs collisions.

Opening pipes

A process can open a pipe with another process using the OPEN_PIPE syscall.

The other process will then receive the RECV_PIPE signal. If no handler is set when the signal is sent, the opening syscall fails.

Pipes' pending data

When a pipe is written to, the data is written to a memory zone. This zone is called the pipe's buffer and it's content is called the pending data.
When a pipe is read from, the pending data is progressively retrieved, erased as the read progresses.

The default size of the pipe's buffer is 64 KB, but this can be extended to up to 16 MB during its creation. When it is reached, no data can be written to the pipe anymore, meaning the other process must read data from it in order to free space to write it.

Pipes locking

When a pipe is being written to or read from, it is locked, which means no other writing or reading can happen during this time. This prevents data races which are a common source of bugs which are complex to debug, while not compromising performances.

Closing pipes

Any of the two processes (be it the receiver or the sender) can close a pipe using the CLOSE_PIPE syscall, providing its SC or RC identifier. The pipe is immediatly closed on both sides, and the other process receives the PIPE_CLOSED signal.

Message pipes

Pipes are designed to transmit streams of data, but sometimes we need to use them to transmit messages. This is why there are specific syscalls and signals to deal with this problem.

A message pipe is a pipe that only sends and receives dynamic-sized messages instead of a stream of bytes.
They have a maximum length of 64 KB, which is the pipes' buffer's minimal capacity. Messages must always be sent at once and cannot be sent partially.
Their length is determined when the message is sent which, coupled to pipes locking, allows to retrieve complete messages directly.

It's not possible to send "non-message" data through a message pipe, as the action of writing to a pipe will automatically check if it's a message pipe and ensure the size and "send at once" requirement are met.

Interactive usage

You can find more informations on interactive usage in the shell specifications.

Service sockets

The downside of message pipes is that they are not designed to handle responses from the receiver process, and they also don't have built-in errors handling.

To solve this, it's possible to use service sockets, which as their name indicate are mainly used to communicate with services. A socket is caracterized by the process opening it, called the service, and the process receiving it, the client.

Opening

Sockets are opened with the OPEN_SERV_SOCK syscall. The other process receives the RECV_SERV_SOCK signal on its side.

Exchanges and messages

Service sockets are based on exchanges, which are sets of messages. An exchange is described by a structure made of the following:

• Identifier (8 bytes)
• Method or notification code (1 byte)
• Status: is the exchange closed, and if so what is the error code (on 2 bytes)
• Message counter for this exchange (4 bytes)

This structure is handled by the kernel itself and can be managed using a set of system calls and signals.

Messages can then be sent using the SEND_SOCK_MSG syscall and received through the RECV_SOCK_MSG signal. To be read, the receiver process must use the READ_SOCK_MSG syscall.

They must be sent at once, like in message pipes. The maximum size is 4 KB by default, but can be extended through the opening syscall up to 256 MB.

A message can either initiate an exchange or answer to an existing one: in the first case, an identifier will be generated by the kernel and returned by the sending syscall, and in the latter the message will receive an incremented identifier that allows to track where in the exchange the message is. This also prevents a process to send two messages before the other one answers.

Concluding exchanges

Exchanges can be either be concluded either by sending an error message or by sending a message with a close indicator to indicate this message will be the last one in the socket and no answer will be accepted.

Sending a message in a concluded exchange will result in a syscall error.

Methods and notifications

An exchange can be opened by a client to request something to the service, it is then called a method. In that case, the message's body is called the arguments.

When the server opens itself an exchange with the client, it's called a notification. A notification message's body is called its data.

Closing service sockets

Service sockets can be closed with the CLOSE_SERV_SOCK syscall, which triggers the SERV_SOCK_CLOSED signal on the other process' side.

Although exchanges are based on signals which are asynchronous by design, the answer mechanism and messages tracking which ensures the receiver process answers before sending another message allows for simplier communications and synchronization between the two processes.

Shared Memory

Shared memory allows a process to share a part of its memory with another process. It has multiple advantages over pipes:

• Data is not copied twice, as the sender process directly shares a part of its own memory
• There is no pipe management, which results in saving operations and less memory accesses
• There is no pipe buffer to manage which means all the data can be sent at once

Its main disadvantage being that all data is shared at once, so there is no synchronization or "asynchronous sending" mechanism, which is the purpose of pipes.

It works by asking the kernel to share the memory through the SHARE_AMS syscall, the target process receiving the RECV_SHARED_AMS signal.

There are two types of sharing:

• Mutual sharing allows both the sharer and the receiver processes to access the shared memory ;
• Exclusive sharing unmaps the shared memory from the sharer and only allows the receiver process to access it

Exclusive mode has several advantages: the sender process to not have to care about managing this memory and avoid overwriting it by accident, but it also ensures the receiving process that the sender will not perform malicious modifications on the shared buffer while the data is processed on its side.

Mutual memory sharing can be stopped using the UNSHARE_AMS syscall, while exclusive sharing are left to the receiver process.

Libraries

Libraries are NightOS' way to share code between multiple applications.

• Manifest

Manifest

As libaries are only meant to share code, there manifest is a lot simplier than applications' manifest.

# [REQ] Informations about the library
infos:
  # [REQ] Library's slug, as stored on the disk in the /apps directory
  slug: sound-synth
  # [OPT] Library's title, as shown when installing it
  name: Sound Synthesizer
  # [REQ] Library's description, as shown when installing it
  description: A small library to synthesize sound through virtual instruments
  # [REQ] Library's license (must be in a list of available licenses)
  license: Apache-2.0
  # [REQ] Library's version, following semantic versioning
  version: 1.0.0
  # [REQ] Library's authors
  authors:
    - name: Clément Nerma
      email: clement.nerma@gmail.com
  # [OPT] Library's icons (in the archive)
  # [OPT] "%{}": either 16, 32, 64, 128 or 256 pixels (icons must be square)
  icon: assets/icons/app/%{}.png

# [REQ] Library package's content
content:
  # Packages can either contain source code only, pre-compiled programs only, or both
  # <for source packages> [REQ]
  source:
    # [REQ] Build tool (must in the list of the toolchain's supported build tools)
    toolchain: rust
    # [REQ] Required build tool-related options
    build: {}
    # [OPT] Build tool-related options
    options:
      optimize: O3

  # <for precomp packages> [REQ]
  precomp: main.nsl

# [REQ] Library dependencies
dependencies:
  sysver: ^1.0.0 # Any stable version

Registry

This document describes the format of the system registy as well as its content.

• Format
  • Notes about types
• HFFR Format
• Structure

Format

The registry is located in the /etc/sys/registry file. It's basically a nested B-Tree map.

Each map has string keys and a value that describes the entry: its purpose, its type with optional constraints, and then the type-safe value.

All types are described as a name, optionally followed by a maximum number of entries between brackets or a maximum number of bytes between braces. Here is the list, in HFRR notation:

The registry's root is a struct. Each key-value association in a struct is called a node. Nodes that are single values (neither a list, a map or a structure) or called leafs.

All sizes that are exceptly provided in bytes must be a multiple of two.

Notes about types

• Durations (time, tmin, tmax and tbtw types) have a precision which indicate the smallest unit of time they accept. 1 is for years, 2 for months, 3 for weeks, 4 for days, 5 for hours, 6 for minutes, 7 for seconds, 8 for milliseconds, 9 for microseconds and 10 for nanoseconds.

• Sizes (size, smin, smax and sbtw types) accept values that are a multiple of their precision. 1 is for terabytes, 2 for gigabytes, 3 for megabytes, 4 for kilobytes, 5 for bytes and 6 for bits.

• For the struct type, there is no need to specify the list of possible keys and the type of associated values, because they are already present in the registry when it's installed by the system - so it's all implicit. The keys in a struct should never change through time, nor the type of the value of each key. As such, it is not possible to create a list of struct, for instance.

• For map structures (structmap and structmapc), the first key when sorting using the default algorithm for each type is guaranteed to never be removed. Its mapped value acts as the model structure for all over values in the map.

HFFR Format

The registry can be converted for debugging to an HFRR text file (Human-Friendly Registry Format).

In this format, structures' keys are described using the key(type): format. Each line following it describing its value is indented by a tabulation (\t). The tabulation is decreased when the value has been fully describes.

Strings are describes using double quotes, with a r prefix symbol for ASCII strings. Floating-point numbers use a . symbol as their decimal separators, and an explicit + or - symbol indicates all numbers' sign.

Each list entry is prefixed by a - symbol, and each key is described as a set of key: value list (on multiple lines for lists and maps).

Empty lists are represented as [] and empty maps as {}.

Durations are represented as a combination of one or more integer numbers each followed by a time suffix (ns for nanoseconds, us for microseconds, ms for milliseconds, s for seconds, m for minutes, h for hours, d for days, w for weeks, mo for months or y for years). Also, combinations use the space separator and numbers are represented with leading zeros if necessary, except for the first number.

So for instance, a duration of 2 days and 10 seconds will give the string 2d 10s.

Sizes are represented just like times, but with different suffixes (TB for terabytes, GB for gigabytes, MB for megabytes, KB for kilobytes, B for bytes and b for bits), and without leading zeroes.

So for instance a size of two terabytes and three kilobyte will be represented as 1TB 3KB.

The easiest way to understand the format is to look at the default registry structure presented below.

Structure

NOTE: This section is under heavy development and is by no mean complete!

Here is the content of the default registry file (when installing NightOS with all default settings), converted to the HFRR format:

# Kernel configuration (only editable in developer mode)
kernel(struct):
  # Signals configuration
  signals(struct):
    # Delay before forced suspension after WILL_SUSPEND signal
    suspend_delay(time(ms)): 500ms
    # Delay before forced termination (kill) after WILL_TERMINATE signal
    terminate_delay(time(ms)): 500ms
    # Delay for a service to answer a connection request
    service_answer_delay(ntime(ms,100ms)): 500ms

# System configuration
system(struct):
  # Debugging options
  debugging(struct):
    # Is development mode enabled?
    dev_mode(bool): true

  # Encryption
  encryption(struct):
    # Is the global storage encrypted?
    global_encrypted(bool): false

  # Date and time
  datetime(struct):
    # Is it based on the internet? (if not, it's a custom one)
    internet_based(bool): true
    # In the case the date is not internet-based, provide a timestamp which indicates the difference between the computer's
    #  local datetime and the one that should be displayed
    custom_based_diff(stime(ns)): 0ns

  # Crash saves
  crash_saves(struct):
    # Restore crash saves as soon as the user logs in
    restore_on_login(bool): true
    # Do not save crash saves for guest users
    disable_for_guest_user(bool): true
    # Delay between each crash save
    collect_every(tmin(s,1s)): 60s
    # Ask an application to create a new crash save even if the previous new has not been completed yet
    collect_if_previous_unanswered(bool): false

  # Freeze-prevention
  freeze_prevention(struct):
    # Reserved amount of memory (0 = disabled)
    reserved_memory(MB): 16MB
    # Reserved amount of CPU per virtual core (0 = disabled)
    reserved_cpu_vcore(uint{1}): 1

# Users (keys = user's unique identifier, value = user's type with 0 = main admin., 1 = admin., 2 = standard, 3 = guest)
# Detailed informations about each user (nickname, privileges, encryption password, ...) and groups are stored
#   in the user profiles file at '/etc/sys/users'
users(map:(uint{4}):(in:(0,1,2,3))):
  0: 1 # System
  1: 1 # Administrator

Vocabulary

This document introduces all vocabulary related to NightOS;

• Graphics-related terms
• Surface
• Interaction
  • Windows
    • Window
    • Borderless windows
    • Resizable windows
    • Windows' title bar
    • Windows' resizing lines

Graphics-related terms

Surface

A surface is a two-dimension area displayed on the screen. It is often used to refer to an application's window.

Interaction

An interaction with a given surface consists in the user clicking on it on pressing a key while this surface is focused.

Windows

Window

A window is a 2D figure provided to an application for manipulation. It does not refer to the desktop environment itself, or to the bootloader's screens for example. There are several types of windows: resizable windows and borderless windows.

Borderless windows

Borderless windows are windows without any titlebar nor resizing lines.

Resizable windows

Resizable windows are the most common type of windows. They feature a title bar as well as resizing lines.

Windows' title bar

A window's title bar is a bar located above the window's content. It contains:

• On its left: a set of four buttons (customizable) which allow to hide the window in the taskbar, restore/maximize it, close it, or suspend/resume the related application ;
• On its center: the window's title, which is controlled by the application and may change during the window's lifetime

Windows' resizing lines

A window's resizing lines are four lines on all sides of a window, 1-pixel wide or high, which allow to extend or reduce the window's size relatively to the line's position.

For example, dragging the top line to a pixel higher on the screen will result in the window being extended to this direction - but other borders of the window won't move. Dragging the top line to the opposite direction will result in the window being compressed to this direction.

The top line is located above the window's titlebar.

Hydre

The shell is the part responsible for running scripts, which are small text-based programs that are natively available on every NightOS installation. It is called Hydre and is part of the system under the form of the sys::hydre service.

You can get a quick overview of Hydre in its technical document.

NOTE: The behaviours described here only apply when using the Pluton terminal application. Although all terminals are expected to follow these specifications as they act as a standard for terminals, they are technically not forced to and so misbehaviours may appear when using an alternative terminal application.

• Shell sessions
  • Sessions security
• Commands evaluation
• Command pipes
• Interactivity
  • Input data
  • User inputs
  • Text output
    • Invalid messages
• Events handling
• Scripting language
  • Pre-evaluation checking

Shell sessions

Before evaluating commands, a shell session needs to be created using the sys::hydre service.

A session has the following properties:

• A (changeable) width and height, in number of characters
• Commands output are concatenated

Commands can then be run inside the created session, and once they are all finished the same session can be closed using the same service. This simplify execution chaining, but also enables commands to be notified through the service's pipes when the session is resized.

Only one shell instruction can be run at a time, but as it may be made of multiple commands (e.g. cmd1 | cmd2), multiple processes may be binded to the same shell session. Also, background commands may be running in parallel of the running command.

For instance, when using the Pluton terminal, it creates on opening a shell session to run the commands in. When a new tab is opened, another shell session is opened for that tab. When the tab is destroyed, the session is destroyed as well, which means all commands running in it (including pipe and background commands) are killed.

Sessions security

When a command is run, the command's PID is registered in the session's actors list. When the command ends, it is removed from the list.
When a process contacts the sys::hydre service to access the session it runs in, Hydre gets the session associated to this process' PID, and returns it. If the PID is not associated to any session, the access is refused and the command won't be able to get any information on any session.

This prevents processes from accessing sessions they aren't part of.

Commands evaluation

Commands are evaluated one by one, as scripts cannot be run in a concurrent way. They are handled as follows:

• Builtin commands are treated internally by the shell
• Application commands will result in launching the requested application in a separate process

Command pipes

When an application is started from a command, its execution context indicates it and the process gets access to several pipes called the command pipes:

The SC/RC identifiers of these pipes are available in the application's context.

The process that launched the command gets the ability to:

• Send data to the callee's CMDIN/CMDUSR pipe ;
• Read data from the callee's CMDMSG/CMDERR/CMDRAW/CMDOUT pipes

As this only applies to processes that can be started from commands, this only applies to application processes ; system processes not being linked to commands and worker processes being started from application processes (and so not directly from commands).

Technically speaking, commands are started by the sys::hydre service and so by a system process. This same service creates the pipes and handles them. For more informations on this, please check the service's documentation.

If the process terminates before the return value has been fully transmitted through CMDOUT or if it closes the CMDOUT pipe before fully transmitting the value, the process is considered as faulty and killed immediatly (if still alive). The calling script (if any) exits with an error message, unless the error is caught with catch, the error message being generated by the system.

Even if the process closes its CMDMSG or CMDRAW pipe properly (by calling the CLOSE_PIPE), the command is not considered as finished until the process itself did not terminate.

Note that when a return value has been fully transmitted through CMDOUT, all pipes are closed and the command is considered as finished.

Interactivity

When an application command is run, the pipes are handled as follows.

Input data

All data coming from a command pipe (|) or from an input pipe (<) (if the command's input type is stream) are transmitted through CMDIN. Once the input data have been fully transmitted, the CMDIN pipe is closed.

User inputs

All user inputs (including raw keystrokes) are transmitted to CMDUSR, except a few ones:

• Ctrl-., which asks the process to suspend (triggers the SUSPEND signal)
• Ctrl-Shift-., which forces the process to suspend (triggers the WILL_SUSPEND signal)
• Ctrl-C, which asks the process to terminate (triggers the TERMINATE signal)
• Ctrl-Shift-C, which forces the process to terminate (triggers the WILL_TERMINATE signal)
• Custom GUI keystrokes like Alt-F4
• System-handled keystrokes like Ctrl+Alt+Del

User input messages use the following format (always starting from the strongest byte/bit):

• Byte 0: modifier keys
  • Bit 0: set if the Command or Windows key is pressed
  • Bit 1: set if the Ctrl key is pressed
  • Bit 2: set if the Alt key is pressed
  • Bit 3: set if the Shift key is pressed
  • Bit 4: set if the Fn key is pressed
  • Bit 5: set if Numeric Lock is enabled
  • Bit 6: set if Caps Lock is enabled
  • Bit 7: set if Scroll Lock is enabled
• Byte 1: keycode
• Bytes 2-5: UTF-8 printable character on 4 bytes, or 0x00 if the character is not printable

Text output

Commands may output either simple text messages (via CMDMSG) or error text messages (via CMDERR).
Conventionnally (but it's up to the terminal application), text messages are displayed by default in white while error messages are displayed in red.

Messages must use the following format (always starting from the strongest byte/bit):

• Byte 0-2: RGB foreground color (0 will use the current one)
• Byte 3-5: RGB background color (0 will use the current one)
• Byte 6: style
  • Bit 0: if set, the message will be displayed in bold
  • Bit 1: if set, the message will be displayed in italic
  • Bit 2: if set, the message will be displayed with a line below it
  • Bit 3: if set, the message will be displayed with a line above it
  • Bit 4: if set, the message will be displayed reversed
• Byte 7: number of control characters
• Byte 7-(N): control characters on 2 bytes each
• Bytes (N+1)-(END): The message itself, encoded in UTF-8 (can be empty)

Control characters use the following format:

• Strongest byte: control character code
  • 0x00: no control character
  • 0x01: move cursor up X times
  • 0x02: move cursor left X times
  • 0x03: move cursor right X times
  • 0x04: move cursor down X times
  • 0x05: move the cursor to the beginning of the line
• Weakest byte: data byte

For instance, an 0x0305 control character is decomposable in the 0x03 code and the 0x05 data byte, which means moving the cursor to the right 5 times.

Invalid messages

A message is considered invalid if at least one of the following conditions is verified:

• The message's length is lower than 8 bytes + 2 * (number of control characters)
• The provided control character is invalid

Messages that do not follow this format will result in displaying Unicode's replacement character (0xFFFD: �) instead.

Messages providing an invalid foreground and/or background color will conventionnally (but it's up to the terminal application) in a specific color to indicate no right color could be determined.

Events handling

Commands can get informations on the current session using the sys::hydre service.

This allows the command to be notified of events like windows resizing. For more informations, see the service's specifications document.

Scripting language

You can find more about the script language in the language's specifications document.

Pre-evaluation checking

Before executing a script, the shell looks for errors in it, such as unknown command, invalid argument, type mismatch and so on. If an error is foud, the script doesn't even start to run ; an error is directly reported and the command is considered as failed.

This prevents errors from happening in the middle of a script, leaving it in an inconsistent state ; this also makes script errors easier to debug as they are reported at compile time.

Shell scripting

The scripting language of Hydre offers a lot of powerful easy-to-use features. This allows to create complex script that are still very readable and maintanable.

• Running a command
  • Combining short arguments
  • Argument values
• Comments
• Variables
• Value types
  • Variables shadowing
• Expressions
• Computing values
  • Mathematical operators
  • Bit-wise operators
  • Logical operators
  • Assignment operators
• Blocks
  • Conditionals
  • Switches
  • Loops
  • Filesystem iteration
  • Variables scoping
• Functions
  • Arguments
  • Return types
  • Methods
  • Failing
    • Retries
    • Global failing
• Nullable types
  • Handle the null value
  • The case of optional arguments
  • Nullable any
• Advanced types
  • Structures
  • Closures
  • Streams
• Data validation
• Event listeners
  • Waiting
• Imports
  • Aliases
  • Import expansions
  • Non-clashing namespace
  • Volatile imports
• Commands input & output
  • Reading a command's output
  • Redirecting the output to a file
  • Output data
    • Reading from CMDIN
    • Returning with CMDOUT
    • Writing to CMDRAW
  • Input of a command
• Running in background
• Environment variables
  • Reading an environment variable
• Commands typing
  • Arguments type
  • Enumerations
  • Return type
  • Conditionals
  • Example
• Native library
  • Utilities
    • env(varname: string) -> any?
    • prompt(message: string) -> string
    • prompt_int(message: string) -> fallible int
    • prompt_float(message: string) -> fallible float
    • confirm(message: string) -> fallible bool
    • choose(options: list[string]) -> fallible int
    • retry_cmd(cmd: command, retries: int) -> fallible
    • exit()
    • last_failed() -> bool
    • rand() -> float
    • rand_int(low: int, up: int) -> fallible int
    • rand_float(low: float, up: float) -> fallible float
  • All types
    • any.str() -> string
  • Nullable types
    • T?.isNull() -> T
    • T?.default(fallback: T) -> T
    • T?.unwrap() -> T
    • T?.expect(message: string) -> T
  • Numbers
    • num.to_radix_str(base: num, leading = false) -> fallible string
  • Characters
    • char.single() -> bool
    • char.codepoints() -> list[int]
    • char.len() -> int
    • char.bytes() -> int
  • Strings
    • string.chars() -> list[char]
    • string.codepoints() -> list[int]
    • string.len() -> int
    • string.bytes() -> int
    • string.parse_int(base = 10) -> fallible int
    • string.parse_float(base = 10) -> fallible float
    • string.upper_case() -> string
    • string.lower_case() -> string
    • string.reverse() -> string
    • string.concat(right: string) -> string
    • string.split(str: string, sep: string) -> string
  • Lists
    • list[char].stringify() -> str
    • list[T].get(index: int) -> T?
    • list[T].expect(index: number, message: string) -> T
    • list[T].unshift(value: T)
    • list[T].push(value: T)
    • list[T].unshift() -> T?
    • list[T].pop() -> T?
    • list[T].sort(asc = true) -> list[T]
    • list[T].reverse() -> list[T]
    • list[T].len() -> int
    • list[string].join(sep = ",") -> string
    • list[T].concat(another: list[T]) -> list[T]
    • list[T].concat(lists: list[list[T]]) -> list[T]
  • Commands
    • command.run() -> int
    • command.fallible()
    • command.ret_str() -> string
    • command.cmdraw() -> stream
    • command.cmdmsg() -> list[string]
    • command.cmderr() -> list[string]
    • command.output() -> list[string]
  • Streams
    • stream.pending() -> bool
    • stream.size_hint() -> int?
• Examples
  • Guess The Number
• Native commands
  • echo: display a value
  • wt: write a file
  • rd: read a file
  • mkdir: create a directory
  • ren: rename a filesystem item
  • mv: move a filesystem item
  • rm: remove a filesystem item
  • ls: list filesystem items
  • fd: find filesystem items
  • syml: manage symlinks

Running a command

Hydre uses an intuitive syntax. Commands are written like they would be in *-sh shells: the command name is followed by arguments, each separated by a space.

Arugments can either be positional (they are written directly), shorts (prefixed by a - symbol and one-character long), or longs (prefixed by two - symbols). Here is an example:

cmdname "pos1" "pos2" -a --arg1

This line runs the command called cmdname, provides two positional arguments pos1 and pos2, a short one a and a long one arg1. Short and long arguments are called dash arguments.

Combining short arguments

It's possible to combine multiple short arguments in once by writing them one after the other:

cmdname -abc

This line is strictly equivalent to:

cmdname -a -b -c

Argument values

Short and long arguments can also require a value. This value must be provided using an = symbol or by simply using a space:

cmdname -s=1 --long=2
# or:
cmdname -s 1 --long 2

Comments

Comments can be written on a single line with the # symbol:

cmdname # single-line comment

Everything after the # symbol is ignored. For multiple-line comments, it's required to use three # symbols:

###
this
is a
multi-line
comment
###

Variables

Variables are declared with the var keyword:

var age = 19

Here, we declare a variable age with value 19. As variables are typed, this variable will only be allowed to contain numbers from now on - no strings, no booleans, nothing else.

And to assign a new value to it:

age = 20

Value types

All arguments must match their expected type: if the command is expecting a number, we can't give it a list for instance.

Here is the list of all types and how they are written:

# Booleans (bool) (`true` or `false`)
_ = true

# Integers (int)
_ = 3

# Floating-point numbers (float)
_ = 3.14

# Characters (char)
_ = 'a'

# Strings (string)
_ = "abc"

# Lists of a given type (list[type])
_ = [ 3, 3.14 ]

# Paths (path) - must contain at least one `/` to indicate clearly that it's a path and not a string or something else
_ = dir/file.ext
_ = ./file.ext
_ = /tmp

# Commands (used to run custom commands later in functions)
_ = @{ command "pos1" -s --long }

A string is composed of multiple chars, which are made of single codepoints. This means a grapheme cluster made of multiple codepoints will need to be encoded in a string.
There is also the num type which accepts integers and floating-point numbers, and any which allows values of all types.
Finally, there is the void type which cannot be written 'as is' but is used in special contexts like commands return values. It's a type that contains no data at all.

There are also presential arguments, which are dash arguments that take no value. The command will simply check if the argument was provided or not.

NOTE: In order to avoid writing errors, positional arguments cannot be provided after a presential argument.

For instance, considering pos1 and pos2 are positional arguments, --pres a presential argument and --val a non-presential long argument:

# VALID
command "pos1" "pos2" --pres --val 2

# VALID
command "pos1" --val 2 "pos2" --pres

# VALID
command "pos1" --pres --val 2 "pos2"

# VALID
command --press --val 2 "pos1" "pos2"

# INVALID (we could think by reading this that "pos2" is the
#          value of the non-presential argument "--pres")
command "pos1" --pres "pos2" --val 2

Variables shadowing

Any variable can, inside a program, be replaced by a new variable with a same name but with a different type. This is called shadowing.

It can be useful when converting data from a type to another, such as:

var names: list[string] = [ "Jack", "John" ]

var names: string = names.join(",") # this is called a _method_, we'll talk about them later

Here, we shadowed the names variable to create a new one with a different type from the data we had before, which means we cannot access the original names variable anymore.

Expressions

To use a variable, we can directly use it like this::

tellage ${age}

So this code:

var age = 20
tellage ${age}

Is equivalent to this one:

tellage 20

It's also possible to perform computations using expressions:

var age = 20
var add = 12
tellage ${age + add}

String and characters can also be inserted inside a string:

var name = "Jack"
echo "Hello, ${name}!" # Hello, Jack!

Values in lists through their index (starting at 0):

var names = [ "Jack" ]
echo "Hello, ${names[0]}!" # Hello, Jack!

Note that getting an out-of-bound index will make the program panic, which means it exits immediatly with an error message.

var names = [ "Jack" ]
echo "Hello, ${names[1]}!" # Panics

Computing values

It's also possible to compute values using operators. Each operator takes one or multiple operands, which can be either a variable or a literal value.

Mathematical operators

Mathematical operators all take two operands. If any of the operands is not a number, it will fail. If both operands are integers, the result will be an integer, but if at least one is a floating-point number, so will be the result.

The operators are:

• +: addition
• -: substraction
• *: multiplication
• **: pow
• /: division
• //: floating-point division (gives a floating-point number even if the two operands are integers)
• %: remainder (works only with two integers)

Bit-wise operators

Bit-wise operators only take integer operands and produce an integer result:

• & bit-by-bit and
• | bit-by-bit or
• ^ bit-by-bit exclusive or
• << binary left shift operator
• >> binary right shift operator
• ~ one's complement - takes a single number

Logical operators

Logical operators take two operands and return a boolean:

Finally, there is the ! operator, which takes a single operand on its right, and simply reverts a boolean.

Assignment operators

The neutral assignment operator = can be prefixed by any mathematical, bit-wise or logical operator's symbol(s). The operator's left operand will be the current variable's content, while the right one will be the value on the left of the assignment operator. The result will then be stored in the variable.

Here is an example:

var a = 3

a += 1
a *= 8
a /= 2

echo ${a} # 16

Note that the result's type must be compatible with the variable:

var a = 0

a &&= 1 # ERROR: Cannot assign a 'bool' to an 'int'

There are also the ++ and -- operators, which respectively increase and decrease the desired variable:

var a = 0

a ++
a ++
a --

echo ${a} # 1

Blocks

Blocks allow to run a piece of code multiple times or if a specific condition is met. They are useful combined to comparison operators.

Conditionals

Conditionals uses the following syntax:

if # condition
  command1
end

When this block is ran, if condition (which is an expression that must result in a boolean) is equal to true, command1 is ran. Here is an example:

if 2 + 2 == 4
  command1
end

It's possible to specify multiple commands at once:

if 2 + 2 == 4
  command1
  command2
  command3
end

Note that all commands must be indented by one tabulation.

It's also possible to run a set of commands in case the condition isn't met too using else:

if 2 + 2 == 4
  command1
else
  command2
end

Finally, conditions can be chained using elif:

if 1 + 1 == 4
  echo "Bizarre."
elif 1 + 1 == 3
  echo "Bizarre!"
else
  echo "Normal."
end

Switches

A switch allows to perform actions depending on a value. It's roughly equivalent to a combination of multiple if and elif statements.

switch rand_int(0, 10)
  when 0
    echo "It's zero!"

  when 1
    echo "It's one!"

  else
    echo "It's something else"
end

Note that, for blocks that only contain a single instruction, we can shorten this using the following syntax:

switch rand_int(0, 10)
  when 0 => echo "It's zero!"
  when 1 => echo "It's one!"
  else   => echo "It's something else!"
end

Loops

Loops allow to run a piece of code for a while. The most common loop is the range loop:

for i in 0..10
  command ${i}
end

This will run command 0 to command 9. To include the upper bound, we must add an = symbol:

for i in 0..=10
  command ${i}
end

This will run command 0 to command 10.

We can also iterate on a list:

var str = "Jack"

var list = [ "Jack", "John" ]

for name in list
  echo ${name}
end

This will display Jack and John.

To get the indexes as well, we can do:

for i, name in list
  echo "${i}: ${name}"
end

This will display 0: Jack and 1: John.

There is another type of loop, which runs a piece of code while a condition is met:

while # condition
  command
end

If the condition is false when the loop is reached, the command will not be ran once. Else, it will be ran as long as the condition is true.

Note that loops can be broke anytime using the break keyword:

for i in 0..10
  command ${i}

  if i == 2
    break
  end
end

This will run command 0, command 1 and command 2 only.

Filesystem iteration

It's possible to iterate on a list of files and directories:

for file in (./*.txt)
  echo "Found a text file: ${file}"
end

The pattern between parenthesis must be a glob pattern. Recursivity is supported to:

for file in (**/*.txt)
  echo "Found a text file: ${file}"
end

Variables scoping

When a variable is declared, it is scoped to the current block, meaning it doesn't exist outside of the current block:

# This variable is declared in the "global" block
# so it's available everywhere in the current script
var firstName = "Jack"

if firstName == "Jack"
  # This variable is declared in an "if" block
  # so it's not available outside of it
  var lastName = "Sparrow"
end

echo ${firstName} # Prints: "Jack"
echo ${lastName} # ERROR ("lastName" is not in scope)

Also, variables are not shared between scripts.

Functions

Functions allow to split the code in several parts to make it more readable, as well as to re-use similar pieces of code across the script.

fn hello()
  echo "Hello!"
end

Now we can call hello this way:

hello() # Will print "Hello !"

Arguments

Function can also take arguments, which must have a type.

fn hello (name: string)
  echo "Hello, ${name}!"
end

hello("Jack") # Will print "Hello, Jack!"

Arguments can be made optional by providing default values. This also allows to get rid of their explicit type as it's now implicit:

fn hello (name = "Unknown")
  echo "Hello, ${name}!"
end

hello()       # Will print "Hello, Unknown!"
hello("Jack") # Will print "Hello, Jack!"

Note that a function's arguments do not require to wrap the value between ${...} as it's implicit. Which means we can write:

var name = "Jack"
hello(name) # Prints: "Hello, Jack!"

We can also combine functions and blocks, for instance:

fn greet (names: list[string]) -> int
  # .len() is called an _extension_, we will see more about that later
  if names.len() == 0
    echo "No one to greet :|"
  else
    for name in names
      echo "Hello, ${name}!"
    end
  end
end

hello([])               # No one to greet :|
hello(["John", "Jack"]) # Hello, John! Hello, Jack!

Return types

Functions can also return values. In such case, they must specify the type of values they return, and ensure all code paths will return a value of this type:

fn add (a: num, b: num) -> num
  return a + b
end

Methods

All value types expose specific functions that can be used with a dot after a variable of the given type, called methods:

var letters = "abcdef"
var letters = letter.split(",")

Here, we use the split method of the string type, which returns a list[string].

Failing

Functions can also fail to indicate something went wrong:

fn divide (a: num, b: num) -> fallible num
  if b == 0
    fail "Cannot divide by 0!"
  else
    return a / b
  end
end

The fallible keyword must be present before the return type to indicate the function may fail (even if the function doesn't return anything).

When a function fails, the program stops and print the provided error message. But it's also possible to handle the error:

fn handle_bad_div (a: num, b: num) -> num
  catch divide(a, b)
    ok result
      echo "Divided successfully: ${a} / ${b} = ${result}"

    err errmsg
      echo "Division failed :("
      echo "Here is the error message: ${errmsg}"
  end
end

Note that you may handle only the success or error case depending on your needs ; you do not have to handle both cases.

This keyword also allows to catch errors from commands:

# Run a command and get error messages from CMDERR instead if the command fails
catch $(somecommand)
  ok data => echo "Success: ${data}"
  err msg => echo "Errors: ${msg.join("; ")}"
end

Retries

It's possible to retry a function until it succeeds using the retry keyword:

fn may_fail ()
  if rand() > 0.5
    fail "I don't like high numbers"
  end
end

retry may_fail()

This will run may_fail, and run it again if it fails, until it succeeds.

It's also possible to specify a maximum number of retries:

retry(5) may_fail()

For information, here is the declaration of the native retry_cmd command, which allows to try to run a command until it succeeds:

fn retry_cmd(cmd: command, retries: string) -> fallible
  retry(retries) cmd()
  if status() != 0
    fail "Command did not suceed after ${retries} retries."
  end
end

It can be used like this:

retry_cmd(@{ read "file.txt" }, 10)

Global failing

A whole script can fail using this keyword, which will result in displaying the error message in CMDERR and exiting immediatly.
The failure may be handled using fallible in the caller script.

Nullable types

Sometimes, it's useful to be able to represent a value that may be either something or nothing. In many programming languages, "nothing" is represented as the null, nil or () value.

Nullable types are suffixed by a ? symbol, and may either contain a value of the provided type or null. Here is an example:

fn custom_rand() -> int?
  var rnd = rand_int(-5, 5)

  if rnd > 0
    return rnd
  else
    return null
  end
end

To declare a variable with an nullable type, we wrap its initialization value in the nullable operator ?(...):

var a = 1 # int
var b = ?(1) # int?

Or we explicitly give the variable a nullable type:

var b: int? = 1 # int?

To initialize the variable with the null value instead, we must use the explicit version:

var c: int? = null # int?

Note that imbricated types are not supported, which means we cannot create int?? values for instance.

Handle the null value

If we try to access an nullable value "as is", we will get a type error:

var a = ?(1)

var b = 0
b = a # ERROR: Cannot use an `int?` value where `int` is expected

We then have multiple options. We can use one of the nullable types' function:

var a = ?(1)

var b = 0

# Make the program exit with an error message if 'a' is null
b = a.unwrap()

# Make the program exit with a custom error message if 'a' is null
b = a.expect("'a' should not be null :(")

We can also detect if a value is null by using the .isNull() method:

var a = ?(1)
var b: int? = null

echo ${a.isNull()} # false
echo ${b.isNull()} # true

There is also the .default(T) method that allows to use a fallback value in case of null:

var a = ?(1)
var b: int? = null

echo ${a.default(3)} # 1
echo ${b.default(3)} # 3

We can also use special syntaxes in blocks:

var a = ?(1)

if some a
  # While we are in this block, 'a' is considered as an 'int'
else
  # While we are in this block, 'a' is considered as 'null'
end

while some a
  # Same here
end

Also, if the program exits in all cases when the argument is considered as null or non-null, the opposite type will be applied to the rest of the program:

var a = ?(1)

if some a
  exit
end

# 'a' is considered as 'null' here

var b = ?(1)

if none b
  exit
end

# 'b' is considered as 'int' here

The case of optional arguments

When a command takes an optional argument, it's possible to provide a nullable value of the same type instead:

var no_newline = ?(false)

echo "Hello!" -n ${no_newline}

If the value is null, the argument will not be provided. Else, it will be provided with the non-null value.

Nullable any

The any type covers any type of values, meaning it accepts absolutely every single value, except the null value. Indeed, any is not nullable by default, so to make it accept null values we must use any? instead.

Advanced types

Structures

Structures map one or multiple fields to as many values. Here is an example:

fn sayHello(person: struct { firstName: string, lastName: string })
  echo "Hello, ${person.firstName} ${person.lastName}!"
end

sayHello({ firstName: "Bat", lastName: "Man" }) # Prints: "Hello, Bat Man!"

In such a simple example, it's easier to directly use a firstName and lastName parameters instead of a struct, but they are useful when returning heterogenous sets of data, for example in a list:

fn listRecursively(dir: path) -> list[struct { name: path, size: int }]
  var list: list[struct { name: path, size: int }] = []

  for item in $(ls ${dir} --details)
    if item.isDirectory
      listRecursively(dir)
    else
      list[] = { name: item.path, size: stats.sizeOf }
    end
  end

  return list
end

But, as this is not very readable, it's better to use a type alias:

type fsItem = struct { name: path, size: int }

fn listRecursively(dir: path) -> list[fsItem]
  var list: list[fsItem] = []
  # ...
end

Closures

Closures are anonymous functions which are generally used to repeat the same group of operations.

var test: fn (string, int) = { a, b -> echo "${a.repeat(b)}" }
test("Hello world! ", 3) # Prints: "Hello world! Hello world! Hello world! "

Note that closures can also return values implicitly:

var test: fn (string, int) -> string = { a, b -> a.repeat(b) }
echo ${test("Hello world!", 3)} # Prints: "Hello world! Hello world! Hello world! "

Also, closures are not forced to take their declared parameters:

type testType = fn (string, int)

var test: testType = { a, b, c -> ### ... ### } # NOT VALID
var test: testType = { a, b    -> ### ... ### } # Valid
var test: testType = { a       -> ### ... ### } # Valid
var test: testType = {         -> ### ... ### } # Valid

Here is a concrete usage example:

fn forEachFile(dir: path, callback: fn (path))
  for item in $(ls ${dir} --details)
    if item.isFile
      callback(item.fullPath)
    end
  end
end

forEachFile(./, { file -> echo "File: ${file}" })

Streams

An usual type for manipulating large data is stream, which is notably used to treat a chunk of data that is either too large for the memory or is more easier to treat as things progress.

Data validation

Some commands may return a value whose type cannot be predicted before runtime. For instance, a command fetching a JSON object from a remote server will, in case of success, return a value but whose type cannot be known beforehand.

Data validation is a feature allowing to scripts to check the type of an unknown value at runtime, using type assertions.

Here is an example:

var test: any = [ 2, 3, 4 ]

# 1. Here, "test" is considered to be of type "any"

if test is list[num]
  # 2. Here, "test" is considered to be of type "list[num]"
else
  # 3. Here, "test" is considered to be of type "any"
end

The isnt keyword can also be used:

var test: any = [ 2, 3, 4 ]

# 1. Here, "test" is considered to be of type "any"

if test isnt list[num]
  # 2. Here, "test" is considered to be of type "any"
else
  # 3. Here, "test" is considered to be of type "list[num]"
end

If, in an "isnt" conditional, the script exits/fails in all cases (or returns if we're in a function), the value is considered with the asserted type for the rest of the script:

var test: any = [ 2, 3, 4 ]

# 1. Here, "test" is considered to be of type "any"

if test isnt list[num]
  # 2. Here, "test" is considered to be of type "any"
  exit
end

# 3. Here, "test" is considered to be of type "list[num]"
# > Because it's impossible for "test" to not be "list[num]" as in that case the program would have exited

This can be used to check complex structures as well:

# Considering we have a `fn fetchJson(url: string) -> any' function that fetches a JSON value from the web

type User = struct { id: num, firstName: string, lastName: string, email: string }

var json = fetchJson("https://mysuperapi.../users/all")

if json isnt User
  fail "JSON doesn't have the correct structure"
end

# "json" is considered as a "User" here

Event listeners

Scripts can listen to events using the on keyword:

on keypress as keycode
  echo "A key was pressed: ${keycode}"
end

This program will display a message each time a key is pressed.

If the event listener is registered in a function, the is automatically unregistered when that function returns. If it's registered outside a function, it is unregistered when the script ends.

fn test()
  on keypress as keycode # (1)
    # Do something
  end
  # Event listener (1) is unregistered here
end

on keypress as keycode # (2)
  # Do something
end

echo "Hello world!"
# Event listener (2) is unregistered here

Waiting

Scripts may wait for a specific event before continuing. This can be achieved without a while loop that consumes a lot of CPU, using the wait keyword:

wait condition

The script will block while the provided condition is not true. The checking interval is defined by the system, and the condition should as fast to check as possible to consume as little CPU as possible.

echo "Please press the <F> key to validate your choice"

var validated = false

on keypress as keycode
  if keycode == KEY_F
    validated = true
  end
end

wait validated

echo "Thanks for validating your choice :D"

It's also possible to wait for a variable to not be null:

echo "Please press a key"

var key: int? = null

on keypress as keycode
  key = keycode
end

wait some key

echo "You pressed key: ${key}"

Imports

As you may already know, command names can be quite long and complicated. In order to prevent from having to repeat very long names that are not really readable, it's recommanded to use imports which are declared at the beginning of script with the form <dev>::<app>::<command>:

import system::fs::read_file

read_file # ...

It's then possible to use the read_file without prefixing.

To perform multiple imports at once:

import system::fs::{read_file, write_file}

It's also possible to only bind the application itself:

import system::fs

fs::read_file # ...

Aliases

Imported commands can also be aliased:

import system::fs as sysfs

sysfs::read_file # ...

Import expansions

It's possible to import all commands from an application, with:

import system::fs::*

read_file # ...

But also to import all applications from a developer, with:

import system::*

fs::read_file # ...

Note that, if a name clash occurs - if two applications or commands with the same name are imported -, the script won't be able to run.

Non-clashing namespace

The non-clashing namespace is a namespace that can be imported, where live all commands whose name is unique across all applications.

For instance, let's imagine we have two applications:

• AppA by DevA, which exposes a cmd_a and a cmd_z command ;
• AppB by DevB, which exposes a cmd_b and a cmd_z command

What happens here? While the cmd_z command has a name clash between AppA and AppB, the cmd_a and cmd_b commands don't. This results in these last two commands being also put in the nonclashing namespace, which can then be imported like a traditional application:

import shell::nonclashing

nonclashing::cmd_a # OK
nonclashing::cmd_b # OK
nonclashing::cmd_z # ERROR (not in namespace)

This means we can also import all commands that don't clash with other ones:

import shell::nonclashing::*

cmd_a # OK
cmd_b # ...
cmd_z # ERROR (not in namespace)

Volatile imports

As volatile applications' commands are not exposed globally, there is a special import syntax for such applications, allowing to import their commands directly from their application package:

import ./app.nva::super_command
super_command # ...

# OR
import ./app.nva as app
app::super_command # ...

Commands input & output

Reading a command's output

Commands output data through pipes. There are several output pipes that can be used:

• CMDOUT is the default output pipe, which returns typed values (the command declares its output type beforehand)
• CMDRAW allows to send a stream, which is useful when dealing with a lot of data or with external data
• CMDMSG allows to send string messages that are displayed in the terminal's windows
• CMDERR allows to send string messages that are also displayed, but as error messages

There is a specific syntax to get the output from each pipe. To get the (typed) output from CMDOUT:

_ = $(echo "Hello!") # Contains: "Hello!"

This is called the typed reception operator. It can be used like this for instance:

echo ${$(echo "Hello!")} # Prints: "Hello!"

But, as this syntax is not very readable, evaluating a single command can be made without the ${...} expression wrapper:

echo $(echo "Hello!") # Prints: "Hello!"

Note that this only work if the command supports piping through CMDOUT.

To get the result from CMDRAW instead (as a stream), if the command supports it:

# '-b' makes 'echo' read from a stream
echo -b $@(streamify "Hello!") # Prints: "Hello!"

To get the output of CMDMSG instead (as a list[string]):

echo $?(echo "Hello!") # Prints: "["Hello!"]"

To get the output of CMDERR (as a list[string]):

echo $!(echo "Hello!") # Prints "[]"

To get the combined output of CMDMSG and CMDERR (as a list[string]):

echo $*(echo "Hello!") # Prints "["Hello!"]"

Note that using the $(...) operator will make the program panic if the command exits with a non-zero status code.

Redirecting the output to a file

It's also possible to redirect the output of a command to a file, using the > operator. The values are converted to strings before being written, except stream values which are written as they are.

echo "Hello!" > ./test.txt

This works because echo outputs by default to CMDOUT, not to CMDMSG. If it did, we could still perform the redirection this way:

echo "Hello!" ?> ./test.txt

The prefixes are the same as for the $(...) operator:

• > for CMDOUT
• @> for CMDRAW
• ?> for CMDMSG
• !> for CMDERR
• *> for CMDMSG and CMDERR combined

Output data

If a script is declared as a command, it gets its own CMDIN, CMDOUT and CMDRAW pipes (the CMDUSR, CMDMSG and CMDERR pipes remain as usual).

They can be accessed using three built-in commands: cmdin, cmdout and cmdraw.

Reading from CMDIN

The cmdin command simply writes in its CMDOUT the value provided in the shell's CMDIN. For instance:

# Considering this shell script accepts `string` values as input.

# If this shell script is called with 'Hello world!' as an input:
echo $(cmdin | length) # Prints: 12

The original type is preserved, which means we can perform typed operations on the input value.

Returning with CMDOUT

The cmdout command takes a typed value and writes it to the shell script's CMDOUT pipe. This also makes the program exit.

Writing to CMDRAW

The cmdraw command takes a stream value and writes it to the shell script's CMDRAW pipe. Only one stream can be piped at a time, so if cmdraw is called while another is pending, the command will simply fail (this can be caught with catch).

Input of a command

Some commands accept inputs through the command pipe | operator. They can be used this way:

echo "Hello world!" > ./somefile.txt

read ./somefile.txt # Prints: "Hello world!"

read ./somefile.txt | length # Prints: 12

How does this work exactly? First, read reads the file and outputs it to CMDOUT as a string, which is then passed to length which happens to accept strings as an input. It then computes the length of the provided input and writes it to CMDOUT, as a number. Which means we can do that:

echo ${ $(read ./somefile.txt | length) * 2 } # Prints: 24

The input may be typed and only accept specific types of values. For instance length only accepts strings, so if we try to give it something else:

echo $(pass 2 | length) # ERROR

What happens here is that we use the builtin pass command which writes to CMDOUT the exact same value we gave it as an input. Then we give it to length, which fails because it doesn't accept numbers.

There's also a shorthand syntax for providing a file's content as CMDIN to a command:

echo $(length < ./somefile.txt) # Prints: 24

For commands that only accept string inputs, the file is automatically decoded and converted to a string. Else, it's kept as a stream.

Running in background

It's possible to run multiple commands in parallel by using background commands. A background command runs, as the name suggests, in the background, and so its output isn't visible. If it fails, it won't generate any error nor affect the status() code.

To run a command in backgroud, we use the bg keyword:

bg hello = sleep 5 -x { i -> echo "Counter: ${i}" }

This will declare an hello variable and put an int value inside it, which is the background command's identifier (BGID). The command will be started and run in parallel of the current program. For instance, the following program:

bg hello = sleep 5 -x { i -> echo "Counter: ${i}" } --end { -> echo "Counter completed!" }

for i in 1..=5
  sleep 1
  echo "Loop: ${i}"
end

echo "Loop completed!"

Will print:

Counter: 1
Loop: 1
Counter: 2
Loop: 2
Counter: 3
Loop: 3
Counter: 4
Loop: 4
Counter: 5
Loop: 5
Counter completed!
Loop completed!

It's possible to wait for a background command by making it run back in the foreground:

bg hello = sleep 5 # The command runs in background
fg hello # The commands comes back to the foreground
         # The program pauses until it completes like for a normal command

It's also possible to let the command run even when the program finishes with detach:

bg hello = sleep 5
detach hello

Or to stop the background command with kill:

bg hello = sleep 5 -x { i -> echo "Counter: ${i}" }

sleep 3
kill hello
echo "Killed."

### Output:
Counter: 1
Counter: 2
Counter: 3
Killed.
<Program stops>###

Environment variables

While traditional variables are always scoped, environment variables are variables that are provided to the script when it starts, and are forwarded to all scripts the main scripts calls, recursively.

They are mostly used to share configuration between programs.

They are usually set:

• Globally, using Central
• For the terminal, using the terminal application's settings
• For the current script, by providing them when running the script

The third case is the most common, here is what it looks like:

# Without environment variables
./myscript.ns

# With environment variables
with MESSAGE="Hello world!" run ./myscript.ns

The environment variable will then be provided to ./myscript.ns, and will also be available in all scripts this script calls.

Reading an environment variable

Environment variables cannot be accessed like traditional variables, they must be retrieved through the env builtin function:

# In "myscript.ns"

var message = env("MESSAGE") # any?

As the variable may not be defined, the function returns a nullable value, so we must check if the variable is indeed defined:

var message = env("MESSAGE")

if none message
  fail "MESSAGE environment variable was not provided"
end

Now we are sure that message is defined, we get an any value, because we don't know the type of the environment variable. So we must use type assertions for that:

var message = env("MESSAGE")

if none message
  fail "MESSAGE environment variable was not provided"
end

if message isnt string
  fail "MESSAGE environment variable is not a string"
end

# Here, "message" is a string

Perfect! Note that, if you want to check if the environment variable exists and is of a specific type at the same time, you can skip the first checking, which is only here to perform specific actions in case the environment variable isn't even defined:

var message = env("MESSAGE")

if message isnt string
  fail "MESSAGE environment variable was not provided or is not a string"
end

# Here, "message" is a string

Commands typing

For script files to be called as commands, they must define a main function and declare a command description.

Here is an example of command description:

cmd
  author "Me <my@email>" # Optional
  license "MIT" # Optional
  help "A program that repeats the name of a list of person"
  return void
  args
    # Declare a positional argument named 'names' with a help text
    pos "names"
      type list[string]
      help "List of names to display"
      optional

      # If this argument is omitted, the command will expect the list of names to be provided through CMDIN
      if absent
        cmdin list[string]
      end

    # Declare a dash argument named 'repeat'
    dash "repeat"
      type int
      short "r"
      long "repeat"
      optional

    # Get the time this command took to complete
    presential "duration"
      short "d"
      long "duration"

      # Conditional return type
      # 'present()' also accepts an optional argument name to check if another argument is present
      if present
        return int
      end
end

Arguments type

Arguments type can be any existing type, or:

• anystr: accepts any type of argument except stream, which will be converted to a string when the command is called (which means the argument will be a string from the command's point of view)

Enumerations

The enum type for arguments indicate the argument only accepts a subset of values (whose type is inferred), which must be specified as a constant. This means the caller cannot use a variable as this argument's value, because the return type may depend on it.

It may also be used as a list of custom values by wrapping the enumeration into a list[...].

Return type

The command's return type can be any existing type.

The options for each argument are:

• type: Required, the type of the argument (nullable types are forbidden)
• help: A help message indicating what the argument does
• short: Short name for a dash argument
• long: Long name for a dash argument
• optional: Indicate the optional can be omitted (the type will be converted to a nullable one)
• default: Make the value optional, but with a default value (so the type will not be nullable)
• requires: Indicate one or several other arguments are required to use this dash one
• conflicts: Indicate this dash argument cannot be used when one or several other specific arguments are already in use
• enum: Allow only a subset of values

For dash arguments, at least short or long must be provided. Also, optional and default cannot be provided at the same time. For presential arguments, at least short or long must be provided. Also, type, optional, default and enum are not accepted.

Conditionals

Conditions can also use the elif and else keywords, and use the present() and absent() operators as well as usual relational operators like == or < for constant values like enums.

Example

Here is an example that uses all these options:

cmd
  args
    # ...
    dash "repeat"
      type int
      help "How many times to repeat the names"
      short "r"
      long "repeat"
      default 1
      requires "arg1"
      conflicts "arg2" "arg3"
      enum list[1, 2, 3, 4]
    end
  end
  # ...
end

The main function takes arguments with the same name as described in the cmd block, and in the same order:

fn main(names: list[string], repeat: int?)
  for i in 0..=repeat.default(1)
    echo ${names.join(", ")}
  end
end

The script can then be called like any command, with the default $(...) operator returning the script's return value:

./myscript.ns ["Jack", "John"] -r 1
# or
var result = $(./myscript.ns ["Jack", "John"] -r 1)

Also, know that scripts can fail too. This allows errors to be handled when the script is run as a function:

# main(names: list[string], repeat: int?) -> fallible

catch $(myscript ["Jack", "John"])
  ok  _ => echo "Everything went fine :)"
  err _ => echo "Something went wrong :("
end

Native library

The native library is a list of functions that are provided by the shell.

All types have extensions, which are functions that can be called using the . symbol, like my_list.extension().

Utilities

env(varname: string) -> any?

Get an environment variable.
Returns null if the environment variable cannot be found.

prompt(message: string) -> string

Ask the user a string.

prompt_int(message: string) -> fallible int

Ask the user an integer number. Fails if the provided input is not a number. Fails if the shell is not interactive.

prompt_float(message: string) -> fallible float

Ask the user a floating-point number. Fails if the provided input is not a floating-point number. Fails if the shell is not interactive.

confirm(message: string) -> fallible bool

Ask the user to confirm a message using an [Y/n] prompt. Fails if the shell is not interactive.

choose(options: list[string]) -> fallible int

Ask the user to pick a value from a list and get the index of the chosen value. Fails if the shell is not interactive.

retry_cmd(cmd: command, retries: int) -> fallible

Run a command and retry it a given number of times if it fails. Fails if the command still fails after all allowed tries.

exit()

Make the program exit.

last_failed() -> bool

Check if the previous command failed.
Returns 0 if no command was ran since the beginning of the script.

rand() -> float

Generate a random floating-point number between 0 and 1.

rand_int(low: int, up: int) -> fallible int

Generate a random integer between low and up. Fails if low is not strictly less than up.

rand_float(low: float, up: float) -> fallible float

Generate a floating-point number between low and up. Fails if low is not strictly less than up.

All types

any.str() -> string

Turns the provided value into a string, depending on the value's type:

_ = (true).str()    # true
_ = (3).str(3)      # 3
_ = (3.14).str()    # 3.14
_ = ('B').str()     # B
_ = ("Yoh").str()   # Yoh
_ = @{ command --arg1 -c 2 -d=4 }.str() # command --arg1 -c 2 -d 4

_ = ["a","b"].str() # [ "a", "b" ]
_ = @{ streamify "Hello world!" }.str() # "<stream>"

Nullable types

T?.isNull() -> T

Check if the value is null.

var a: int? = 1
var b: int? = null

echo ${a.isNull()} # false
echo ${b.isNull()} # true

T?.default(fallback: T) -> T

Use a fallback value in case of null:

var a: int? = 1
var b: int? = null

echo ${a.default(3)} # 1
echo ${b.default(3)}

T?.unwrap() -> T

Make the program exit with an error message if the value is null.

var a = ?(0) # int?
var b = a.unwrap() # int

T?.expect(message: string) -> T

Make the program exit with a custom error message if 'a' is null

var a = ?(0) # INt?
var b = a.expect("'a' should not be null :(") # int

Numbers

num.to_radix_str(base: num, leading = false) -> fallible string

Represent the provided number in a given base. Fails if the base is not between 2 and 36.

_ = (11).to_radix_str(16, false) # "A"
_ = (11).to_radix_str(16, true)  # "0xA"

Characters

char.single() -> bool

Indicate if a character is made of a single codepoint.

char.codepoints() -> list[int]

Get the codepoints composing a character.

char.len() -> int

Get the number of codepoints composing a character.

char.bytes() -> int

Get the size of a character, in bytes.

Strings

string.chars() -> list[char]

Get the characters composing a string.

string.codepoints() -> list[int]

Get the codepoints composing a string.

string.len() -> int

Get the number of codepoints composing a string.

string.bytes() -> int

Get the size of a string, in bytes.

string.parse_int(base = 10) -> fallible int

Tries to parse the provided string as a number in the provided base. Leading zeroes are accepted. 0 symbol followed by b, o, d or x is accepted for bases 2, 8, 10 and 16 respectively. Fails if the string does not represent a number in this base.

_ = ("2").parse_int()   # 2
_ = ("A").parse_int()   # <FAIL>
_ = ("A").parse_int(16) # 11

string.parse_float(base = 10) -> fallible float

Identical to string.parse_int(base) but with floats.

string.upper_case() -> string

Convert a string to uppercase.

_ = ("aBc").upper_case() # "ABC"

string.lower_case() -> string

Convert a string to lowercase.

_ = ("aBc").lower_case() # "abc"

string.reverse() -> string

Reverse a string.

_ = ("abc").reverse() # "cba"

string.concat(right: string) -> string

Concatenate two strings (equivalent to "${left}${right}").

_ = ("a").concat("b") # "ab"

string.split(str: string, sep: string) -> string

Split a string into a list.

split("ab", "")  # [ "a", "b" ]
split("a b", " ") # [ "a", "b" ]

Lists

list[char].stringify() -> str

Turns a list of characters to a string.

_ = ([ 'a', 'b', 'c' ].str() == "abc") # true

list[T].get(index: int) -> T?

Try to get an item from the list, without panicking if the index is out-of-bounds.

var names = [ "Jack", "John" ]

names.get(0) # "Jack"
names.get(1) # "John"
names.get(2) # null

list[T].expect(index: number, message: string) -> T

Get an item from the list, and panic with a custom error message if the index is out-of-bounds.

var names = [ "Jack", "John" ]

names.get(2, "Third item was not found")

list[T].unshift(value: T)

Insert a new value at the beginning of the list.

list[T].push(value: T)

Push a new value at the end of the list.

list[T].unshift() -> T?

Remove the first value from the list and return it.

list[T].pop() -> T?

Remove the last value from the list and return it.

list[T].sort(asc = true) -> list[T]

Sorts a list.

_ = [ 2, 8, 4 ].sort()      # [ 2, 4, 8 ]
_ = [ 2, 8, 4 ].sort(false) # [ 8, 4, 2 ]

list[T].reverse() -> list[T]

Reverse a list.

_ = [ 2, 8, 4 ].reverse() # [ 4, 8, 2 ]

list[T].len() -> int

Count the number of entries in a list.

_ = [ 2, 8, 4 ].len() # 3

list[string].join(sep = ",") -> string

Join a list of strings with a separator.

join([ "a", "b" ])       # "a,b"
join([ "a", "b" ], "; ") # "a; b"

list[T].concat(another: list[T]) -> list[T]

Concatenate two lists.

_ = [ 1, 2 ].concat([ 3, 4 ]) # [ 1, 2, 3, 4 ]

list[T].concat(lists: list[list[T]]) -> list[T]

Concatenate multiple lists.

_ = [ 1, 2 ].concat([ [ 3, 4 ], [ 5, 6 ] ]) # [ 1, 2, 3, 4, 5, 6 ]

Commands

command.run() -> int

Run the command and gets its status code after exit.

command.fallible()

Run the command and fail if the status code after exit is not 0.
Equivalent to calling the command with simple parenthesis like cmd().

command.ret_str() -> string

Run the command and get its stringified return value.

command.cmdraw() -> stream

Run the command and get its CMDRAW output.

command.cmdmsg() -> list[string]

Run the command and get its CMDMSG output.

command.cmderr() -> list[string]

Run the command and get its CMDERR output.

command.output() -> list[string]

Run the command and get its CMDOUT and CMDERR outputs combined.

Streams

stream.pending() -> bool

Check if the stream is still pending. If the pipe is complete (which means if its pipe is closed), false will be returned.

stream.size_hint() -> int?

Get the stream's size hint. If no size hint was provided for this stream, null will be returned.

Examples

Guess The Number

while true
  var won = false
  var secret = rand_int(0, 100)

  echo "Secret number between 0 and 100 has been chosen."

  while !won
    var user_input = retry prompt_int("Please input your guess: ")

    if user_input < secret
      echo "It's higher!"
    elif user_input > secret
      echo "It's lower!"
    else
      echo "You guessed the number \\o/!"
      won = true
    end
  end

  tries = retry(5) confirm("Do you want to play again?")

  if !(retry(5) confirm("Do you want to play again?"))
    echo "Goodbye!"
    break
  end
end

Native commands

The native commands are commands that are available in every program without import.

echo: display a value

Display a value to CMDOUT.

# echo [-n] { <anystr> | -b <stream>}
#
# "-b": print a stream as UTF-8
# "-n": don't put a newline break at the end of the content

echo "Hello world"
echo -n "Hello world!"
echo -b $(streamify "Hello world!")

wt: write a file

Write a content to a file.

# wt [-a] [-n] [-c] <path> <anystr>
#
# "-a": append to the end of the file
# "-n": don't append a newline symbol at the end of the content
# "-c": fail if the file doesn't exist instead of creating it

Note that sometimes it can be clearer to use a redirection pipe:

# Overwrite file
echo "Hello world!" > ./test.txt

# Append to file
echo "Hello world!" >> ./test.txt

rd: read a file

Read a file as a string value.

# rd [-s] [--stream-size <int>] <path>
#
# "-s": read a stream instead of reading the full content
# "--stream-size": specify the size of the stream when "-s" is provided (rounded to higher pipe buffer multiplier)

mkdir: create a directory

Create a new directory.

# mkd [-s] <path>
#
# "-s": fail if the directory already exists

ren: rename a filesystem item

Rename a filesystem item.

# re [-m] <oldname: path> <newname: path>
#
# "-m": move if the 'newname' is located inside another directory

mv: move a filesystem item

# mv [-n] <file: path> <dest_dir: path>
#
# "-n": create the target directory if it does not exist

rm: remove a filesystem item

Remove a filesystem item.

# rm [-r | --recursive] [-n | --non-empty] [-t | --trash] {<path> | -l <list[path]>}
#
# "-r": allow removing empty directory
# "-n": allow removing even non-empty directories
# "-l": remove a list of paths
# "-t": move the item to the user's trash

ls: list filesystem items

List filesystem items.

# ls [-t | --tree] [-r | --recursive] [-h | --hidden] [--file-only | --dir-only] [<path>]
#
# "-t": display as a tree (implies "-r")
# "-r": list recursively
# "-h": list hidden files
# "--file-only": only display files
# "--dir-only": only display directories

fd: find filesystem items

Find filesystem items matching provided criterias.

# fd [-t | --types list["dir" | "file" | "symlink" | "device"]]
#    [-a | --absolute]
#    [-L | --follow]
#    [-e | --extension <string>]
#    [-n | --name <string>]
#    [-r | --regex <string>]
#    [-i | --ignore-case]
#    [-x | --exec <command>] <path>
#
# "-t": only list items of a given type (by default, only files and symlinks are shown)
# "-a": list absolute file paths instead of relative ones
# "-L": follow symbolic links
# "-e": only list files with the provided extension (directory will be excluded)
# "-n": only list items whose name contain the provided string (`^` and `$` can be used for indicating items must start or end by it)
# "-r": only list items matching a provided POSIX regex
# "-i": ignore case (requires "-n" or "-r")
# "-x": execute a command for each found item

syml: manage symlinks

Manage symbolic links.

# sl {-r | --read} <path>: check a symlink's target
#
# sl [-u | --update] <srcpath> <targetpath>: create a symlink
#
# "-u": update the symlink to the new target path if it already exists instead of failing

Signals

Signals are a type of KPC. They are used by the kernel to send informations to processes about a specific event.

• Technical overview
• List of signals
  • 0x01 HANDLER_FAULT
  • 0x02 MEM_FAULT
  • 0x10 SUSPEND
  • 0x11 WILL_SUSPEND
  • 0x12 TERMINATE
  • 0x13 WILL_TERMINATE
  • 0x20 RECV_PIPE
  • 0x21 PIPE_CLOSED
  • 0x26 RECV_SERV_SOCK
  • 0x27 RECV_SOCK_MSG
  • 0x29 SERV_SOCK_CLOSED
  • 0x2A SERVICE_CONN_REQUEST
  • 0x2B SERVICE_CLIENT_CLOSED
  • 0x2C SERVICE_CLIENT_QUITTED
  • 0x2D SERVICE_SERVER_QUITTED
  • 0x33 READ_BACKED_AMS
  • 0x34 WRITE_BACKED_AMS
  • 0x35 RECV_SHARED_AMS
  • 0x37 UNSHARED_AMS

Technical overview

When a process is created, the kernel associates it:

• A signals handler table (SHT) ;
• A signals queue ;
• A readiness indicator

Each signal has a 8-bit code that identifies it, as well as a 32 bytes datafield which is used to attach additional informations about the signal.

When the kernel sends a signal to a process, it first checks if an handler is already running. If so, it simply pushes the signal to the queue.

Else, it checks the readiness indicator. If it is false (so if the process did not sent the READY syscall yet), the signal is pushed to the queue.

Else, it checks in the SHT if the signal has a handler. If there is no handler, depending on the specific signal, it may either be ignored or use a default behaviour (this is documented for each signal).

If a handler is found, the kernel checks if the pointer points to a memory area that is executable by the current process. If it isn't, the signal is converted to an HANDLER_FAULT one. If the signal that was being sent was already an HANDLER_FAULT, the process is killed.

The kernel then switches the process to its main thread and makes it jump to the handler's address, then resumes it.

When the handler returns (or the default behaviour completes), if the signal was expecting an answer, the kernel reads it from specific registries and does whatever it needs to do. Then, itchecks if the signals queue is empty. If it is, the kernel simply makes the process jump back to the address it was to before the signal was emitted, and switch to the original thread.

Else, it interrupts the process again and proceeds to treat the first signal on the queue after removing it.

On the performances side, signals use interrupts, meaning the process' current tasks are instantly interrupted to let it handle the signal without delay. Also, the datafield and answer are provided through CPU registers, avoiding memory accesses.

List of signals

You can find below the exhaustive list of signals.

0x01 HANDLER_FAULT

Sent when a signal is sent to a process but the registered handler points to a memory zone that is not executable by the current process. If the sending of this signal to the process results to another fault, it's called a double handler fault and the process is immediatly killed.

If no handler is registered for this signal, it will kill the process when received.

Datafield:

• Faulty signal ID (8 bytes)

0x02 MEM_FAULT

Sent when the process tried to perform an unauthorized access on a memory address.

Datafield:

• Faulty address (8 bytes)
• Access error (1 byte):
  • 0x01: tried to read memory
  • 0x02: tried to write memory
  • 0x03: tried to execute memory

0x10 SUSPEND

Sent when the process is asked to suspend. It's up to the process to either ignore this signal or suspend itself using the SUSPEND syscall.

0x11 WILL_SUSPEND

Sent when the process is asked to suspend. If it is not suspended after the provided delay, the process is suspended.

Datafield:

• Registry's system.signals.suspend_delay key (default: 500ms) (2 bytes)

0x12 TERMINATE

Sent when the process is asked to terminate. It's up to the process to either ignore this signal or terminate itself (preferably by using the EXIT syscall).

0x13 WILL_TERMINATE

Sent when the process is asked to terminate. If it does not terminate by itself before the provided delay, the process is killed.

If no handler is registered for this signal, it will kill the process when received.

Datafield:

• Registry's system.signals.terminate_delay key (default: 2s) (2 bytes)

0x20 RECV_PIPE

Sent to a process when another process of the same application and running under the same user opened an pipe with this process, giving it the other part.
The command code can be used to determine what the other process is expecting this one to do. This code does not follow any specific format.

Datafield:

• Pipe creator's PID (8 bytes)
• Pipe creator's application's ANID (4 bytes)
• Pipe SC or RC identifier (8 bytes)
• Command code (2 bytes)
• Pipe identifier type (1 byte): 0x00 if the pipe identifier is an RC, 0x01 if it's an SC
• Mode (1 byte): 0x00 if it's a raw pipe, 0x01 if it's a message pipe
• Size hint in bytes (8 bytes), with 0 being the 'no size hint' value

0x21 PIPE_CLOSED

Sent to a process when a pipe shared with another process is closed.

NOTE: This does not apply to service pipes.

Datafield:

• Closing type (1 byte):
  • 0x00 if the pipe was closed properly using the CLOSE_PIPE syscall
  • 0x01 if the other process brutally terminated |
• Pipe identity (1 byte): 0x00 if this process contained the RC part, 0x01 if it contained the SC part (1 byte)
• RC or SC identifier (8 bytes)

0x26 RECV_SERV_SOCK

Sent to a process when another process opened a service socket with this one.

Datafield:

• Service socket creator's PID (8 bytes)
• Service socket creator's application's ANID (4 bytes)
• Service socket identifier (8 bytes)
• Size of the buffer, multiplied by 4KB (2 bytes)

0x27 RECV_SOCK_MSG

Sent to a process when a message has been sent through a service socket.
To read the message, the process must use the READ_SOCK_MSG syscall.

Datafield:

• Servive socket identifier (8 bytes)
• Exchange identifier (8 bytes)
• Exchange method (1 byte)
• Size of the message (4 bytes)
• Status (1 byte):
  • Bit 0: set if this message did create a new exchange
  • Bit 1: set if this message is an error message
  • Bit 2: set if this message closed the socket

0x29 SERV_SOCK_CLOSED

(was there a message sent before that closed the socket)

Sent to a process when a pipe shared with another process is closed.

NOTE: This does not apply to service pipes.

Datafield:

• Closing type (1 byte):
  • 0x00 if the pipe was closed properly using the CLOSE_PIPE syscall
  • 0x01 if the other process brutally terminated |
• Pipe identity (1 byte): 0x00 if this process contained the RC part, 0x01 if it contained the SC part (1 byte)
• RC or SC identifier (8 bytes)

0x2A SERVICE_CONN_REQUEST

Sent to a service process' dispatcher thread when another process tries to etablish a connection through the CONNECT_SERVICE syscall.

The process is expected to answer using the ACCEPT_SERVICE_CONNECTION under the provided delay, else it's considered as a rejection.

If no handler is registered for this signal, it will kill the process when received.

NOTE: This signal cannot be received if the application does not expose a service.

Datafield:

• Callee process' ID (8 bytes)
• Connection's unique request ID (8 bytes)
• Command code (2 bytes)
• Registry's system.signals.service_answer_delay key (default: 1000ms) (2 bytes)

0x2B SERVICE_CLIENT_CLOSED

Sent to a client thread to indicate its client closed before the connection was properly terminated. The thread is expected to terminate as soon as possible (there is no time limit though).

0x2C SERVICE_CLIENT_QUITTED

Sent to a client thread to indicate its client asked to close the connection. The associated RC and SC are immediatly closed.

0x2D SERVICE_SERVER_QUITTED

Sent to a process that previously established a connection with a service, to indicate the associated service thread closed before the connection was properly terminated.

Datafield:

• Connection's unique request ID (8 bytes)

0x33 READ_BACKED_AMS

Sent to a process when a signal-backed abstract memory segment (AMS) is accessed in read mode.

Datafield:

• AMS ID (8 bytes)
• Relative address accessed in the segment (8 bytes)
• Access mode (1 byte): 0x00 for read, 0x01 for execution

Expected answer:

• Associated data for this file (4 bytes)
• Page fault (1 byte):
  • 0x00: no page fault
  • 0x01: address is out-of-range
  • 0x02: hardware fault

0x34 WRITE_BACKED_AMS

Sent to a process when a signal-backed abstract memory segment (AMS) is accessed in write mode.

Datafield:

• AMS ID (8 bytes)
• Relative address accessed in the segment (8 bytes)
• Data to write (4 bytes)

Expected answer:

• Page fault (1 byte):
  • 0x00: no page fault
  • 0x01: address is out-of-range
  • 0x02: hardware fault

0x35 RECV_SHARED_AMS

Sent to a process when an abstract memory segment (AMS) is shared by another process.

Datafield:

• Sender PID (8 bytes)
• Command code (2 bytes)
• AMS ID (8 bytes)
• Sharing mode (1 byte): 0x00 for mutual sharing, 0x01 for exclusive sharing
• Access permissions (1 byte):
  • For mutual sharings: strongest bit for read, next for write, next for exec
  • For exclusive sharings: 0b11100000

0x37 UNSHARED_AMS

Sent to a process when anabstract memory segment (AMS) is unshared by the sharer process.

Datafield:

• Unsharing type (1 byte):
  • 0x00 if the shared memory was unshared properly using the UNSHARE_AMS syscall
  • 0x01 if the other process brutally terminated

System calls

System calls, abbreviated syscalls, are a type of KPC. They allow a process to ask the kernel to perform an action.

• Technical overview
• List of syscalls
  • 0x01 HANDLE_SIGNAL
  • 0x02 UNHANDLE_SIGNAL
  • 0x03 IS_SIGNAL_HANDLED
  • 0x04 READY
  • 0x10 GET_PID
  • 0x12 SUSPEND
  • 0x13 EXIT
  • 0x20 OPEN_PIPE
  • 0x21 SEND_PIPE
  • 0x22 PIPE_WRITE
  • 0x23 PIPE_READ
  • 0x24 PIPE_INFO
  • 0x25 CLOSE_PIPE
  • 0x26 OPEN_SERV_SOCK
  • 0x27 SEND_SOCK_MSG
  • 0x28 READ_SOCK_MSG
  • 0x29 CLOSE_SERV_SOCK
  • 0x2A CONNECT_SERVICE
  • 0x2B END_SERVICE_CONN
  • 0x2C ACCEPT_SERVICE_CONN
  • 0x2D REJECT_SERVICE_CONN
  • 0x30 MEM_ALLOC
  • 0x31 MEM_FREE
  • 0x32 VIRT_MEM_AMS
  • 0x33 BACKED_AMS
  • 0x34 DEVICE_AMS
  • 0x35 SHARE_AMS
  • 0x36 AMS_SHARING_INFO
  • 0x37 UNSHARE_AMS
  • 0x38 MAP_AMS
  • 0x39 UNMAP_AMS
  • 0x3A SET_DMA_MEM_ACCESS
  • 0xA0 EXECUTION_CONTEXT
  • 0xD0 PROCESS_ATTRIBUTES
  • 0xD1 SET_PRIORITY
  • 0xD2 ENUM_DEVICES
  • 0xD3 DEVICE_INFOS

Technical overview

Syscalls are performed using CPU interruptions to notify the kernel.

A syscall is made of a 8-bit code, as well as up to 8 arguments with up to 64-bit value each.
When performing a syscall, the process will put in a specific CPU register an address poiting to a memory address containing in a row the syscall's code and its arguments. For most syscalls, code and arguments will be not be longer than 128 bits, but some may use larger arguments.

Some syscalls require the process to send a buffer of data. In such case, the process simply provides a pointer to the said buffer - so the argument's size will vary depending on the length of memory addresses.

After preparing the syscall's code and arguments, the process raises a specific exception that is caught by the kernel. When the syscall is complete, the kernel puts the result values in specific registers and resumes the process. This means that all syscalls are synchronous.

System calls always return two numbers: a 8-bit one (errcode) and a 8 bytes one (return value). If the errcode is not null, then an error occured during the syscall. The specific value indicate the encountered type of error:

• 0x00: cannot read syscall's code or arguments (error while reading memory)
• 0x01: the requested syscall does not exist
• 0x02: at least one argument is invalid (e.g. providing a pointer to the 0 address)
• 0x03: unmapped memory pointer (e.g. provided a pointer to a memory location that is not mapped yet)
• 0x04: memory permission error (e.g. provided a writable buffer to an allocated but non-writable memory address)

Errors are encoded this way:

• 0x00 to 0x0F: generic errors (see above)
• 0x10 to 0x1F: invalid arguments provided (e.g. value is too high)
• 0x20 to 0x2F: arguments are not valid in the current context (e.g. provided ID does not exist)
• 0x30 to 0x3F: resource errors (e.g. file not found)
• 0x40 to 0xFF: other types of errors

System calls' code are categorized as follows:

• 0x00 to 0x0F: signal handling
• 0x10 to 0x1F: process management
• 0x20 to 0x29: pipes
• 0x2A to 0x2F: services communication
• 0x30 to 0x3F: memory management
• 0xA0 to 0xAF: applications-related syscalls
• 0xD0 to 0xDF: reserved to system services

Note that advanced actions like permissions management or filesystem access are achieved through the use of IPC.

List of syscalls

You can find below the exhaustive list of system calls.

0x01 HANDLE_SIGNAL

Register a signal handler.
If the address pointed by this syscall's is not executable by the current process when this signal is sent to the process, the signal will be converted to an HANDLER_FAULT signal instead.

Arguments:

• Code of the signal to handle (1 byte)
• Pointer to the handler function (8 bytes)

Return value:

Empty

Errors:

• 0x10: The requested signal does not exist

0x02 UNHANDLE_SIGNAL

Unregister a signal handler, falling back to the default signal reception behaviour if this signal is sent to the process.

Arguments:

• Code of the signal to stop handling (1 byte)

Return value:

Empty

Errors:

• 0x10: The requested signal does not exist
• 0x20: The requested signal does not have an handler

0x03 IS_SIGNAL_HANDLED

Check if a signal has a registered handler.

Arguments:

• Code of the signal (1 byte)

Return value:

• 0 if the signal is not handled, 1 if it is (1 byte)

Errors:

• 0x10: The requested signal does not exist

0x04 READY

Indicate the system this process has set up all its event listeners, so it can start dequeuing signals.

NOTE: When this signal is sent, all queued signals will be treated at once, so the instructions following the sending of this signal may not be ran until quite a bit of time in the worst scenario.

WARNING: Signals will not be treated until this syscall is sent by the process!

Arguments:

None

Return value:

Empty

Errors:

• 0x20: The process already told it was ready

0x10 GET_PID

Get the current process' PID.

Arguments:

None

Return value:

• Current process' PID (8 bytes)

Errors:

None

0x12 SUSPEND

Suspend the current process.

Arguments:

None

Return value:

• Amount of time the process was suspended, in milliseconds (8 bytes)

Errors:

• 0x20: the current process is not an application process

0x13 EXIT

Kill the current process.

A SERVICE_CLIENT_CLOSED signal is sent to all services connection the process has.
If the current process is a service, a SERVICE_SERVER_QUITTED signal is sent to all active clients.

Arguments:

None

Return value:

None (never returns)

Errors:

None

0x20 OPEN_PIPE

Open a pipe with a process of the same application and running under the same user and get its SC.
The buffer size multiplier indicates the size of the pipe's buffer, multiplied by 64 KB. The default (0) falls back to a size of 64 KB.
The command code can be used to indicate to the target process which action is expected from it. It does not follow any specific format.
The target process will receive the SC/RC's counterpart through the RECV_PIPE signal, unless notification mode states otherwise.

Arguments:

• Target process' PID (8 bytes)
• Command code (2 bytes)
• Pipe type (1 byte): 0x00 to create a write pipe, 0x01 to create a read pipe
• Buffer size multiplier (1 byte)
• Transmission mode (1 byte): 0x00 to create a raw pipe, 0x01 to create a message pipe
• Notification mode (1 byte): 0x00 to notify the process with the RECV_PIPE signal, 0x01 to skip it
• Size hint in bytes (8 bytes), with 0 being the 'no size hint' value

Return value:

• Pipe SC identifier (8 bytes)

Errors:

• 0x10: Invalid transmission mode provided
• 0x11: Invalid notification mode provided
• 0x20: The provided PID does not exist
• 0x21: The target process is not part of this application
• 0x22: The target process runs under another user
• 0x23: Notification mode is enabled but the target process does not have a handler registered for the RECV_PIPE signal

0x21 SEND_PIPE

Share an RC or SC identifier with another process.
This will trigger in the target process the RECV_PIPE signal, unless the notification mode tells otherwise.

When the target process writes through the received SC or read from the received RC, the performance will be equal to writing or reading through the original RC/SC identifier.

Arguments:

• Pipe RC or SC identifier (8 bytes)
• Target PID (8 bytes)
• Notification mode (1 byte): 0x00 to notify the process with a pipe reception signal, 0x01 to skip the signal

Return value:

None

Errors:

• 0x10: Notification mode is enabled but the target process does not have a handler registered for the RECV_PIPE signal

0x22 PIPE_WRITE

Write data through a pipe.
Messages will always be sent at once when writing to message pipes.
If the data is 0-byte long, this pipe will return successfully without waiting, even if the target pipe's buffer is full or locked.

Arguments:

• Pipe SC identifier (8 bytes)
• Number of bytes to write (4 bytes)
• Pointer to a readable buffer (8 bytes)
• Mode (1 byte): 0x00 = block until there is enough space to write, 0x01 = fail if there is not enough space to write or if the pipe is locked, 0x02 = write as much as possible

Return value:

Encoded on 4 bytes:

• If mode is 0x00: remaining capacity of the pipe
• If mode is 0x01: 0x00 if the cause of failure was because the pipe was locked, 0x01 if it was because of of a lack of space in the target buffer
• If mode is 0x02: number of bytes written

Errors:

• 0x10: Invalid mode provided
• 0x20: The provided SC identifier does not exist
• 0x21: The provided SC was already closed
• 0x22: The provided SC refers to a message pipe but the provided size is larger than 64 KB
• 0x23: The provided SC refers to a message pipe but the 0x02 mode was provided
• 0x30: There is not enough space in the pipe to write all the provided data and the mode argument was set to 0x01

0x23 PIPE_READ

Read pending data or message from a pipe.
If the pipe was closed while the buffer was not empty, this syscall will still be able to read the remaining buffer's data - but the pipe will not be able to receive any additional data. Then, once the buffer is empty, the pipe will be made unavailable.

Arguments:

• Pipe RC identifier (8 byte)
• Mode (1 byte): 0x00 = block until there are enough data to read, 0x01 = fail if there is not enough data to read or if the pipe is locked, 0x02 = read as much as possible
• Number of bytes to read (4 bytes): 0 = read as much data as possible
• Pointer to a writable buffer (8 bytes)

Return value:

Encoded on 4 bytes:

• If mode is 0x00: remaining bytes in the buffer
• If mode is 0x01: 0x00 if the cause of failure was because the pipe was locked, 0x01 if it was because of of a lack of space in the target buffer
• If mode is 0x02: number of read bytes

Errors:

• 0x10: Invalid mode provided
• 0x20: The provided RC identifier does not exist
• 0x21: The provided RC was already closed
• 0x22: There is no pending data in the pipe and the mode argument was set to 0x01
• 0x23: The provided RC refers to a message pipe but the 0x02 mode was provided

0x24 PIPE_INFO

Get informations on a pipe from its RC or SC identifier.

Arguments:

• Pipe RC or SC identifier (8 bytes)

Return value:

• Status (1 byte):
  • Bit 0 (strongest): indicates if the pipe is opened
  • Bit 1: indicates if the pipe is a message pipe
  • Bit 2: indicates if the pipe's buffer is full
  • Bit 3: indicates if the pipe is locked
  • Bit 4: indicates if a writing request is pending (waiting for the pipe to be unlocked)
  • Bit 5: indicates if a reading request is pending (waiting for the pipe to be unlocked)
  • Bit 6: indicates if the provided identifier is an SC
• Pipe's buffer's capacity (8 bytes)
• Remaining data before the pipe's buffer is full (8 bytes)
• Pipe's creator's PID (8 bytes)

Errors:

None

0x25 CLOSE_PIPE

Close a pipe properly. The RC and SC parts will be immediatly closed.
The other process this pipe was shared with will receive the PIPE_CLOSED signal unless this pipe was created during a service connection.
If this syscall is not performed on a pipe before the process exits, the other process will receive the same signal with a specific argument to indicate the communication was brutally interrupted.

Arguments:

• Pipe RC or SC identifier (8 bytes)

Return value:

None

Errors:

• 0x10: The provided RC/SC identifier does not exist
• 0x11: The target process already terminated
• 0x20: The provided RC/SC identifier is part of a service pipe

0x26 OPEN_SERV_SOCK

Open a service socket.
Triggers the RECV_SERV_SOCK signal on the receiver process' side.

Arguments:

• Client process PID (8 bytes)
• Buffer size multiplier by 4 KB (2 bytes) - 0 fall backs to 4KB

Return value:

• Socket identifier (8 bytes)

0x27 SEND_SOCK_MSG

Send a message through a service socket exchange.
This syscall can also be used to create a new exchange.

Arguments:

• Socket identifier (8 bytes)
• Exchange identifier (8 bytes) - 0 creates a new exchange
• Method or notification code (1 byte) - non-zero value if not creating an exchange to close it with a non-error message
• Error code (2 bytes)
• Number of bytes to write (4 bytes)
• Pointer to the message's content (8 bytes)

Return value:

• Exchange identifier (8 bytes)
• Message counter for this exchange (4 bytes)

Errors:

• 0x20: Unknown socket identifier
• 0x21: Socket is already closed
• 0x22: Unknown exchange identifier
• 0x23: Exchange has already been concluded

0x28 READ_SOCK_MSG

Read the pending message of a service socket.

Arguments:

• Socket identifier (8 bytes)
• Address of a writable buffer (8 bytes)

Return value:

• Number of written bytes (4 bytes)
• 0x01 if a message was retrieved, 0x00 if none was pending (1 byte)
• Status (1 byte):
  • Bit 0: set if this message did create a new exchange
  • Bit 1: set if this message is an error message
  • Bit 2: set if this message closed the socket

Errors:

• 0x20: Unknown socket identifier
• 0x21: Socket is already closed
• 0x22: Unknown exchange identifier
• 0x23: Exchange has already been concluded

0x29 CLOSE_SERV_SOCK

Close a service socket.
Triggers the SERV_SOCK_CLOSED signal on the receiver process' side.

Arguments:

• Socket identifier (8 bytes)

Return value:

None

Errors:

• 0x20: Unknown socket identifier
• 0x21: Socket is already closed

0x2A CONNECT_SERVICE

Ask a service to etablish connection. The current process is called the service's client.

If the current process already has an active connection (a connection that hasn't been closed) to the target service, it will fail unless the flexible mode argument is set.

NOTE: When this signal is sent, the service's answer will be waited, so the instructions following the sending of this signal may not be ran until several seconds in the worst scenario.

Arguments:

• Target application's ANID (4 bytes)
• Command code (2 bytes)

Return value:

• Unique connection ID (8 bytes)
• Pipe SC identifier (8 bytes)
• Pipe RC identifier (8 bytes)
• Flexible mode (1 byte): 0x00 by default, 0x01 returns the existing connection ID an active connection is already in place with the service

Errors:

• 0x10: Invalid flexible mode provided
• 0x20: The provided ANID does not exist
• 0x21: Target application does not expose a service
• 0x22: Current process already has an active connection to the target service and flexible mode is not set
• 0x30: Failed to send the SERVICE_CONN_REQUEST due to a double handler fault
• 0x31: Service rejected the connection request

0x2B END_SERVICE_CONN

Tell a service to properly close the connection. The associated pipe SC and RC channels will immediatly be closed.

Arguments:

• Unique connection ID (8 bytes)

Return value:

None

Errors:

• 0x10: The provided connection ID does not exist
• 0x20: This connection was already closed
• 0x21: The associated service thread already terminated

0x2C ACCEPT_SERVICE_CONN

Confirm the current service accepts the connection with a client.
A dedicated message pipe's SC and another's RC will be provided to communicate with the client.

This will create a new client thread in the current process, which is meant to be dedicated to this specific client.
The client thread will not receive any SERVICE_CONN_REQUEST signal, only dispatcher thread will.

When the associated client terminates, the SERVICE_CLIENT_CLOSED signal is sent to this thread.

Arguments:

• Connection's unique request ID (8 bytes)

Return value:

• 0x00 if the current process is now the associated client's thread, 0x01 else
• Pipe RC identifier (8 bytes)
• Pipe SC identifier (8 bytes)

Errors:

• 0x10: This request ID does not exist
• 0x20: The process which requested the connection already terminated
• 0x30: Answer was given after the delay set in the registry's system.signals.service_answer_delay key (default: 1000ms)

0x2D REJECT_SERVICE_CONN

Reject a connection request to the current service.

Arguments:

• Connection's unique request ID (8 bytes)

Return value:

None

Errors:

• 0x10: This request ID does not exist
• 0x20: The process which requested the connection already terminated
• 0x30: Answer was given after the delay set in the registry's system.signals.service_answer_delay key (default: 1000ms)

0x30 MEM_ALLOC

Allocate a linear block of memory.

WARNING: Allocated memory will not be rewritten, thus it may contain non-zero data. Therefore the caller process shall ensure memory is used correctly.

Arguments:

• The number of pages to allocate (8 bytes)

Return value:

• Pointer to the newly-allocated block of memory (8 bytes)

Errors:

• 0x30: The kernel could not find a linear block of memory of the requested size

0x31 MEM_FREE

Unallocate a linear block of memory.

Shared memory pages must first be unshared through the UNSHARE_AMS syscall.
Mapped memory pages must be unmapped through the UNMAP_AMS syscall.

WARNING: Memory will not be zeroed, therefore the caller process shall ensure critical informations are zeroed or randomized before freeing the memory.

Arguments:

• Pointer to the start address to unallocate the memory from (8 bytes)
• The number of pages to unallocate (8 bytes)

Return value:

None

Errors:

• 0x10: The provided start address it not aligned with a page
• 0x20: The provided start address is out of the process' range
• 0x21: The provided size, added to the start address, would exceed the process' range
• 0x22: One or more of the provided pages was not allocated (e.g. unmapped page or memory-mapped page)
• 0x23: One or more of the provided pages are shared with another process

0x32 VIRT_MEM_AMS

Create an abstract memory segment (AMS) from a part of the current process' address space.

Arguments:

• Address of the first page to register in the AMS (8 bytes)
• Number of bytes to register (8 bytes)

Return value:

• AMS ID (4 bytes)

Errors:

• 0x10: Start address is unaligned
• 0x11: Number of bytes is unaligned
• 0x22: Address is out of range

0x33 BACKED_AMS

Create an abstract memory segment (AMS) backed by the READ_BACKED_AMS and WRITE_BACKED_AMS signals.

Copy-on-write support can be enabled to allow the receiver process to write data in its own memory space. Written pages will be allocated by the kernel and won't be backed anymore by the READ_BACKED_AMS signal. The backer process won't be able to see these changes, and the WRITE_BACKEND_AMS signal won't be trigerred on its side.

Arguments:

• Length of the AMS (8 bytes)
• Copy-on-write mode (1 byte): 0x00 to disable, 0x01 to enable

Errors:

• 0x10: Invalid COW mode provided
• 0x11: Provided length is unaligned

0x34 DEVICE_AMS

Create an abstract memory segment (AMS) from a device's memory through Mapped Memory Input/Output (MMIO).

Requires the current process to have the device in its drivable devices attribute.

Arguments:

• SDI of the device to map in memory (4 bytes)
• Start address in the device's memory (8 bytes)
• Number of bytes to map (8 bytes)
• Start address to map in this process' memory (8 bytes)

Return value:

• AMS ID (8 bytes)

Errors:

• 0x10: The mapping's start address is not aligned with a page
• 0x11: The mapping's length is not a multiple of a page's size
• 0x12: The mapping's size is null (0 bytes)
• 0x20: The provided device SDI was not found
• 0x21: The provided device is not compatible with MMIO
• 0x22: This device is not registered in this process' drivable devices attribute

0x35 SHARE_AMS

Share an abstract memory segment (AMS) with another process.

This will trigger in the target process the RECV_SHARED_MEM with the provided command code, unless the notification mode states otherwise.

The mutual mode allows both processes to access the memory, with the sharer setting the permissions for the receiver to limit its access. Copy-on-write can also be enabled to allow the receiver process to write data without affecting the sharer process' memory.

The exclusive mode allows, only when sharing AMS made from existing memory pages from its original process, to unmap the original pages from the said process to let the exclusive access to the target process. This is useful when transferring temporarily large chunks of data to another process. Also, access permissions are ignored when using exclusive mode.

Arguments:

• Target process' PID (8 bytes)
• Command code (2 bytes)
• Notification mode (1 byte): 0x00 to notify the process with the RECV_SHARED_AMS signal, 0x01 to skip it
• Mode (1 byte):
  • Mutual: 0 b 0 0 0 0 <1 to enable copy-on-write> <1 to enable read> <1 to enable write> <1 to enable exec>
  • Exclusive: 0 b 0 0 0 0 1 0 0 <1 to unmap original pages>

Return value:

• AMS ID (8 bytes)

Errors:

• 0x10: Invalid notification mode provided
• 0x11: Invalid mode provided
• 0x12: Access permissions were not set but the sharing mode is set to mutual
• 0x13: Access permissions were provided but the sharing mode is set to exclusive
• 0x14: Invalid exclusive mode provided
• 0x30: There is not enough contiguous space in the receiver process' memory space to map the shared memory

0x36 AMS_SHARING_INFO

Get informations about a shared abstract memory segment (AMS).

Arguments:

• AMS ID (8 bytes)

Return value:

• Sharer process' PID (8 bytes)
• Receiver process' PID (8 bytes)
• Sharing mode (1 byte): 0x00 for mutual mode, 0x01 for exclusive mode
• Shared buffer's start address (8 bytes)
• Sharer buffer's length (8 bytes)
• Command code (2 bytes)
• Access permissions (1 byte): for mutual sharings, strongest bit for read, next for write, next for exec ; for exclusive sharings, 0x00

Errors:

• 0x10: Unknwon AMS ID provided

0x37 UNSHARE_AMS

Stop sharing an abstract memory segment (AMS) started by SHARE_AMS. Note that exlusive sharings cannot be unmapped.

This will trigger in the target process the UNSHARED_AMS signal.

Arguments:

• AMS ID (8 bytes)
• PID to stop sharing with (8 bytes) - 0 to stop sharing with all processes

Return value:

None

Errors:

• 0x10: Unknown AMS ID provided
• 0x20: Provided AMS ID is exclusive
• 0x21: Provided AMS was not shared with the provided process

0x38 MAP_AMS

Map an abstract memory segment (AMS) in the current process' address space.

Arguments:

• AMS ID (8 bytes)
• Start mapping address in the AMS (8 bytes)
• Address to map the AMS (8 bytes)
• Number of bytes to map (8 bytes)

Return value:

None

Errors:

• 0x10: Unknown AMS ID provided
• 0x20: Provided mapping address or address+length is out-of-range in the AMS
• 0x21: Provided address to map or address+length is out-of-orange in this process' address space

0x39 UNMAP_AMS

Unmap an abstract memory segment (AMS) from the current process' address space.
If the AMS is mapped at multiple addresses of this process, only one of the mappings will be unmapped by default.

Arguments:

• AMS ID (8 bytes)
• Mapping address (8 bytes) - 0 to unmap from all addresses

Return value:

Empty

Errors:

• 0x10: Unknown AMS ID provided
• 0x20: Provided AMS it not mapped at this address

0x3A SET_DMA_MEM_ACCESS

Allow or disallow a device to access a range of addresses through Direct Memory Access (DMA) in the current process' address space.

Requires the current process to have the device in its drivable devices attribute.

Arguments:

• SDI of the device to map in memory (4 bytes)
• Start address in the current process' address space (8 bytes)
• Length (8 bytes)
• Authorization (1 byte): 0x00 to allow the device to use this range, 0x01 to cancel an authorization

Return value:

None

Errors:

• 0x10: The range's start address is not aligned with a page
• 0x11: The range's length is not a multiple of a page's size
• 0x12: The range's size is null (0 bytes)
• 0x20: The provided device SDI was not found
• 0x21: The provided device is not compatible with DMA
• 0x22: This device is not registered in this process' drivable devices attribute

0xA0 EXECUTION_CONTEXT

Get informations from the application's execution context.

Arguments:

• Information to get (1 byte):

  • 0x00: all the context
  • 0x01: startup reason
  • 0x02: context header
  • 0x03: command-line arguments

• Pointer to a writable buffer (8 bytes)

Return value:

• Number of written bytes (8 bytes)

Errors:

• 0x10: invalid information number provided
• 0x20: caller process is a system service

0xD0 PROCESS_ATTRIBUTES

System service-only syscall.
Get a process' attributes.

Arguments:

For value-based attributes:

• Attribute code (1 byte):

  • 0x00: PID
  • 0x01: Process' priority
  • 0x02: Running user's ID
  • 0x03: Parent application ID
  • 0x04: Execution context (startup reason)
  • 0x05: Execution context (header)
  • 0x06: Execution context (arguments)

• Action code:

  • 0x00: Read the information (followed by a writable address on 8 bytes)
  • 0x01: Write the information (followed by the value to write)

For list-based attributes:

• Attribute code (1 byte):

  • 0x00: Memory mappings
  • 0x01: Permissions
  • 0x02: Drivable devices

• Action code (1 byte) followed by its optional arguments:

  • 0x00: Get the number of elements
  • 0x01: Get the value at a given index => index (4 bytes)
  • 0x02: Update the value at a given index => index (4 bytes) + value (? bytes)
  • 0x03: Insert a value at a given index => index (4 bytes) + value (? bytes)
  • 0x04: Push a value at the end of the list => value (? bytes)
  • 0x05: Remove a value from the list => index (4 bytes)
  • 0x06: Remove the last value from the list
  • 0x10: Check if the list contains a given item => value to look for (? bytes)

• Address to a writable buffer (8 bytes)

Return value:

• Number of written bytes (if applies) (8 bytes)

Errors:

• 0x10: Invalid action code provided
• 0x11: Invalid attribute number provided
• 0x20: Caller process is not a system service
• 0x21: This system service is not allowed to access or edit this attribute
• 0x22: Provided index is out-of-bounds

0xD1 SET_PRIORITY

System service-only syscall.
Set the priority of a process.
If the set priority is different than 0, the kernel won't adjust the priority automatically anymore.
Setting it to 0 will reset it to the kernel's choice.

Arguments:

• Process PID (8 bytes)
• Priority to set (1 byte) with 0 to let the kernel set it automatically

Return value:

None

Errors:

• 0x10: Provided priority is higher than 20
• 0x20: Caller process is not a system service
• 0x21: Provided PID was not found

0xD2 ENUM_DEVICES

System service-only syscall.
List devices matching a provided CII.

For each device, its SDI (4 bytes) is written to the provided address.

Arguments:

• CII of the devices to list (4 bytes) 0 will list all devices
• Address of a writable buffer (8 bytes)

Return value:

• Number of devices found with the provided criterias (4 bytes)

Errors:

• 0x10: Invalid connection type in CII

0xD3 DEVICE_INFOS

System service-only syscall.
Get the raw device descriptor of a single device.

Arguments:

• SDI of the device to get informations from (4 bytes)
• Address of a writable buffer (8 bytes)

Return value:

• Number of written bytes (1 byte)

Errors:

• 0x20: No device was found with this SDI

Kernel

NightOS' kernel is named Cosmos. It is a micro-kernel which tries to be as simple and straightforward as possible, delegating all non-trivial tasks such as filesystem access or permissions management to services.

NOTE: This document is in its very early stages, and so is far from being complete. Major changes may and will be made to related documents.

Documents

• Hardware - how the kernel interacts with hardware
• Kernel-process communication - how the kernel communicate with processes and vice-versa
• Memory - memory organization and management
• Processes - processes concept and management
• Data structures - data structures used by the kernel to represent things in memory

Hardware management

This document describes how the kernel interacts with hardware.

• Hardware detection
• Connection-specific device descriptor
• Connection interface identifier
• Session device identifier
• Raw device descriptor
• Drivers

Hardware detection

Devices are detected during the boot process and then periodically after startup. This allows to hotplug some additional components afterwards.

As all components do not use the same connection protocols, the detection process depends on the connection:

• PCI-Express components are detected through their Configuration Space
• IDE/SATA components are detected through the IDE/SATA controller
• USB components are enumerated through the USB protocol stack

Some components may not be detected through these though, such as some legacy ISA devices, which will be detected through a set of methods like ACPI enumeration or simply checking UART serial ports.

Connection-specific device descriptor

All hardware components (devices) expose a normalized identifier whose format depends on the connection type (PCI-Express, SATA, ...). This identifier is called the connection-specific device descriptor (CSDD).

Its size can vary up to 256 bytes.

Connection interface identifier

The connection interface identifier (CII) is a 4-byte number describing what a component is connected to:

• Connection type (1 byte):
  • 0x01: PCI-Express
  • 0x02: IDE
  • 0x03: SATA
  • 0x04: M.2
  • 0x05: USB
  • 0x06: RGB
• Bus number (1 byte)
• Port number (2 bytes)

For instance, the seventh USB port on the second bus will have the 0x05010006 CII.

Session device identifier

The kernel generates for each device a session device identifier (SDI), which is a random 4-byte number specific to the current session, allowing to plug up to 4 billion devices in a single session.

Raw device descriptor

The raw device descriptor (RDD) is a data structure (up to 264 bytes) made of the followings:

• SDI (4 bytes)
• CII (4 bytes)
• CSDD (up to 256 bytes)

This descriptor is then used by the sys::hw service to expose the device to the rest of the operating system.

Drivers

Drivers and software <-> hardware devices communications are handled by the sys::hw system service.

You can find more about how drivers work in this section.

Kernel-Process Communication

The Kernel-Process Communication (KPC) describes how a process can interact with the kernel, and vice-versa.

There are two types of KPC:

• System calls, which are used by a process to ask the kernel to perform an action ;
• Signals, which are used by the kernel to send informations about an event to a process

Note that, unlike many operating systems like Linux, it's not possible for a process to send a signal to another. Only the kernel is allowed to emit signals.

For more advanced features, like permissions management or filesystem, check IPC.

Memory

This document describes how the kernel organizes and manages the memory.

Pages

TODO

Isolation

Each process has its own 64-bit address space, preventing it from accessing other processes' data. The memory is made of pages, which must be either be allocated or mapped using virtual memory segments.

Abstract memory segments

An abstract memory segment (AMS) is an identifier which refers to a segment of memory which doesn't actually exist. To be used, they must be mapped in a process' memory to be accessed like regular memory. The kernel then intercepts all memory accesses to these mappings and handle them, depending on their nature which cover three cases:

• Mapping existing memory pages to others, or sharing them with other processes ;
• Mapping a device's memory into the process' own memory space ;
• Making a virtual memory space handled by signals

An AMS can then be mapped at multiple places in a process' memory, or shared with other processes. The kernel handles mappings to get optimal performances and reduce the number of memory accesses as much as possible.

Processes

Apart from the kernel itself, all programs run in processes.

• Why processes
• Switching and cycles
• Process attributes
• Performance balancing
  • Automatic priority attribution
• Drivable devices

Why processes

Separating programs in threads presents several advantages:

1. This allows to take advantage of multi-core architectures by running multiple programs in parallel
2. Each program has its own data space and does not share its data with other programs
3. Each process has its own permissions and thus cannot bypass what the user chosen

Switching and cycles

To run processes, the kernel simply iterates over the list of existing processes, and allow them to run a given number of instructions. Then, the control is taken back by the kernel which runs the next process, and so goes on.

This happens as follows:

1. A process is selected
2. If the process is currently suspended, it is ignored
3. Its registers are restored by the kernel (if any)
4. The process runs a given number of instructions
5. The kernel takes back the control of the CPU
6. The process' registers are saved
7. Go to step 1

These steps are known as a cycle.

Process attributes

Each process has a set of attributes which contains critical informations on it (lists are usually PLL):

• PID (8 bytes)
• Priority (1 byte)
• Running user's ID (8 bytes)
• Parent application ID (8 bytes) - 0 for system services
• Pointer to the execution context (8 bytes) - 0 for system services
• PLL(e=32) for memory mappings
• PLL(e=32) for raw permissions
• PLL(e=32) for drivable devices

Performance balancing

Each process has a priority number, between 1 and 20, which indicates how much its performances must be prioritized compare to other processes.

The basics can be found here.

More specifically, the higher the priority of a process is, the faster it will run. Here are the priority-dependant aspects of a process:

• Number of instructions run per cycle
• Priority when accessing I/O through services

Comparatively, when a process has a high priority, other processes will run a tad slower.

Automatic priority attribution

Processes' priority is automatically adjusted by the kernel, unless it is manually assigned through the SET_PRIORITY syscall.

The priority is determined based on multiple factors:

• Does the process have a fullscreen window?
• Does the process owns the active window?
• Does the process owns a visible window?
• Is the process a driver or service? If so, how much is it used?

Drivable devices

The drivable devices attribute contains the list of all devices' SDI the current process can drive.

The goal of this attribute is to determine if the process is allowed to map a device's memory by creating an AMS from it using the DEVICE_AMS syscall, as well as using DMA-related instructions in the CPU.

This attribute is managed by the sys::hw service and can only be updated by this service.

Raw permissions

Raw permissions are used by system services to determine the permissions of a process without sending a message to the sys::perm service and waiting for its answer, which would be costly in terms of performance.

These permissions use a specific structure, specified in the related service's specifications document.

Permissions can be managed using the sys::perm service.

Data structures

This documents describes the structures used by the kernel to represent the data it uses in memory.

Packed linked lists

Packed linked lists (PLL) are linked lists used for items whose size is both small (usually <= 32 bytes) and fixed for all items.

It uses a system of same-size entries, each containing a micro bump allocator.

The goal of a PLL is to provide a blazing fast read and iteration speed, while compromising on insertion and deletion speeds.

Caracteristics

A PLL is caracterized by its item size, length (the number of items in the list) and the number of items per entry (NIE) (up to 255), which is the number of items which can be stored per entry. It is noted PLL(e=<number of items per entry>[, s=<item size in bytes>][, l=<number of elements currently in the list>]).

Structure in memory

Each entry is a contiguous suite of bytes which can store up to NIE items contiguously. It starts by either a pointer to the next entry (on 8 bytes), or the number of items actually initialized in the current entry (pre-filled with zeros to be stored on 8 bytes).

For instance, let's take a PLL(e=3, s=2, l=4). Its content is the four following items:

• 0xDEADBEEF
• 0x01234567
• 0x89ABCDEF
• 0xBEEFDEAD

If the first entry is located at address 0x00001000 and the second at address 0x00002000, here is the PLL's representation in memory with big-endian representation (with _ representing garbade data):

0x0000000000001000: 00 00 20 00 DE AD BE EF 01 23 45 67 89 AB CD EF
0x0000000000002000: 00 00 00 01 BE EF DE AD __ __ __ __ __ __ __ __

As you can see, the first entry contains the address to the next entry, followed by the content of the first three items (contiguously).
The second entry is the last one and so simply contains the number of initialized items, followed by the last item's content, and then garbage as this memory zone is not initialized yet.

Checking an entry's type

To check if the first byte of an entry is the next entry's address or the number of initialized items, we simply have to perform a simple comparison: if the byte is greater than 0xFF (255 in decimal, which is the maximum allowed number of items per entry), then it's an address, else it's a number of initialized elements.

The ratio

A PLL also has a ratio, which is the number of bytes reserved for items in each entry, divided by the total number of bytes. So, in our example, 3 items per entry * 4 bytes = 12 bytes, while the total number of bytes per entry also takes into account the address itself, so 12 bytes + 8 bytes = 20 : our PLL's ratio is 0.6.

This is quite a low ratio, meaning we waste a lot of space. The ratio must be kept as near as 1.0 as possible, while maintaining a reasonable memory footprint for each entry.

Performances bottlenecks

A thing to keep in mind is that PLL have a considerable performances bottleneck: when an entry is filled, the next must be allocated with a size that's a lot larger than a single item's size. That's why, when an entry is full, it should be allocated on the moment time is the less critical.

Performances are especially bad when inserting new items in the list or when removing ones, as many data needs to be moved around.

Regarding updating elements, this requires to write both the item itself. Inserting elements in a not-yet full entry requires to write the element itself as well as incrementing the entry's counter.

Performance advantages

As you can see in the above bottlenecks, PLL are not meant to be performant on writings. They are meant to be fast for reads, especially sequential reads. The higher the NIE of the list, the fastest it will be to find an element in it, or to read every element of the list one by one (iteration).

Also, as the counter is packed with the other data of each entry, it presents a reduced risk of cache miss.

Computing the length is reasonably fast.

Length-first variant

There is a variant of the PLL that stores the total length of the list somewhere in the memory. In that case, the first byte of the last entry does not need to store the number of items in the entry.

This variant is interesting being we can instantly get the number of items in the list, but the downside is that reading sequentially the list will also incur an additional reading to know where the last entry ends, and updating the total length incur a high risk of cache fault as the chances of the memory area where the length is as well as the entry itself be in the cache at the same time are pretty low.

The length-first variant should only be used when accessing the length instantly is critical.

Tricking the NIE

Increasing the NIE will:

• Reduce the needs of allocating
• Fasten iteration times
• Fasten insertion times (when the last entry isn't full)

But also:

• Increase the memory cost (of the last entry)
• Worsen allocation performances (as the entries are larger)

System services

As NightOS' kernel is not monolithic but a microkernel, it only handles the most fundamental tasks of the system, like memory and processes management, as well as direct hardware communication.

The vast majority of its features can be found in the system service, which special services run by the system itself under the sys DID.

This splitting implies that most low-level features of the system are documented in the individual services' specifications documents, which you will find here.

Nomenclature

System services are referred to as the sys:: services.

All methods and notifications describe the required permissions to use them, their arguments.

They also use common error codes:

• 0x00: cannot read syscall's code or arguments (error while reading memory)
• 0x01: the requested syscall does not exist
• 0x02: at least one argument is invalid (e.g. providing a pointer to the 0 address)
• 0x03: unmapped memory pointer (e.g. provided a pointer to a memory location that is not mapped yet)
• 0x04: memory permission error (e.g. provided a writable buffer to an allocated but non-writable memory address)
• 0x05: insufficient permissions
• 0x10 to 0x1F: invalid arguments provided (e.g. value is too high)
• 0x20 to 0x2F: arguments are not valid in the current context (e.g. provided ID does not exist)
• 0x30 to 0x3F: resource errors (e.g. file not found)
• 0x40 to 0xFF: other types of errors

All methods return an answer, though it may be empty (indicated by a None). System services' answers always conclude the exchange.

List of system services

• sys::fs: filesystem management
• sys::hw: hardware communication
• sys::perm: permissions management
• sys::net: network communications
• sys::crypto: cryptography utilities
• sys::crashsave: crash saves management
• sys::flow: flows management
• sys::hydre: Hydre shell service

sys::fs service

Methods

TODO

Notifications

TODO

sys::hw service

The sys::hw service is in charge of hardware devices. It coordinates and manages communications with the hardware.

• Hardware detection
• Device formats
  • Device type descriptor
  • Device identifier
  • Driver device descriptor
  • Driven device type
• Normalization
  • Normalized methods
  • Normalized interrupts
  • Normalized notifications
  • Patterns
• Drivers
  • A note on performances
• Methods
  • 0x01 ENUM_DEVICES
  • 0x02 SUBSCRIBE_DEVICES
  • 0x10 REGISTER_DRIVER
  • 0x11 UNREGISTER_DRIVER
  • 0x20 NOTIFY_PROCESS
  • 0xA0 ASK_DRIVER
• Notifications
  • DEVICE_EVENT
  • DEVICE_INTERRUPT
  • DRIVER_METHOD_REQUEST
  • DEVICE_NORM_NOTIF

Hardware detection

Hardware detection is handled by the kernel itself, which then exposes a raw device descriptor (RDD) as well as a connection interface identifier (CII) and a session device identifier (SDI).

Device formats

This section describes the multiple formats used by this service to deal with devices.

Device type descriptor

From the RDD and CII is derived the device type descriptor (DTD), which describes the device's type. Its composition and size depends on the connection type, but it varies from empty (0 byte) if the connection type guarantees no information, up to 256 bytes.

The format remains to be determined but should be along the lines of a number-based equivalent of ModAlias, like :

• PCI-Express:
  • Vendor (8 bytes)
  • Sub-vendor (8 bytes)
  • Type (8 bytes)
  • Sub-type (8 bytes)
  • ...
• ...

Device identifier

It also derives a unique device identifier (UDI) encoded on 265 bytes, which is made of:

• SDI (4 bytes)
• CII (4 bytes)
• Size of the DTD (1 byte)
• DTD (256 bytes, weakest bits filled with zeros)

Driver device descriptor

A driver device descriptor (DDD) is a data structure meant to be used by drivers. It uses the following format:

• Bytes 000-264: Device's UDI
• Bytes 265-512: Future-proof

Driven device type

A driven device type (DDT), usually referred to as the device type, is generated by the driver for each device is drives. This is a normalized value, used by the system to determine which actions can be performed through this device.

It's a 4-byte value, the strongest two bytes describing the category and the weakest two the sub-category.

The following list contains all possible values for DDTs, but is far from being complete yet. It will also grow over time as new device types appear on the market.

• 0x0001: Storage
  • 0x0001: Hard drive
  • 0x0002: SSD
  • 0x0003: USB flash drive
  • 0x0004: SD flash memory card
• ...

Normalization

Normalized methods

When a device is driven, other processes can ask this service to use normalized methods. These are methods that allow to perform a specific action or to receive normalized notifications about specific events of a specific device.

There are several methods, depending on the device's type (DDT). Notifications differ as well.

The following list contains all methods and related notifications for all DDTs, but is far from being complete yet. It will also grow over time as new device types appear on the market and as existing devices evolve to provide new features.

TODO

Normalized interrupts

Some devices use interrupts to notify the system of a particular event. In such case, the interrupt is normalized to a format called the normalized interrupt format, which is then sent to the driver process using the DEVICE_EVENT notification.

The normalized interrupt format depends on the device's type (DDT).

The following list contains all normalized interrupts for all DDTs, but is far from being complete yet. It will also grow over time as new device types appear on the market and as existing devices evolve to provide new features.

TODO

Normalized notifications

The normalized notifications are a type of notifications sent by a driver to processes that subscribed to them using normalized methods. They are derived from the device's interrupts and events pulled from its memory, after a translation by the driver itself.

The normalized notification format depends on the device's type (DDT).

The following list contains all normalized notifications for all DDTs, but is far from being complete yet. It will also grow over time as new device types appear on the market and as existing devices evolve to provide new features.

TODO

Patterns

Several methods of this service use patterns, which allow to match devices depending on several criterias.

A pattern is a data structure whose size varies from 5 to 277 bytes made of the following:

• Pattern (1 byte)
  • Bit 0: match all connection types
  • Bit 2: match all buses
  • Bit 3: match all ports
• Connection type (1 byte)
• Bus number (1 byte)
• Port number (1 byte)
• DTD length (1 byte) - 0 to omit DTD
• DTD pattern indicator (16 bytes, only if DTD) - indicates which bytes of the DTD must be used as patterns
• DTD (up to 256 bytes)

It's possible to match only devices that use a given connection type, and more specifically on a given bus and/or port.

It's also possible to list only devices that match a specific DTD pattern. For that, the bit corresponding to the byte number in the DTD pattern indicator must be set.

For instance, providing the DTD 0x0100B2 with the DTD pattern indicator set to 0b01000000, the second byte will match all devices.

Drivers

From a higher level point of view, drivers are services that declare themselves as being able to handle certain type of devices through the REGISTER_DRIVER method, using patterns.

When a device is connected, using multiple criterias which are yet to be determined, a driver is selected from the list of drivers able to handle this specific device. This driver process then receives the DEVICE_EVENT notification.

From this point, the driver can create an AMS from the device's memory using the DEVICE_AMS syscall.

It can also get informed of interrupts the device raises through the DEVICE_INTERRUPT notification.

Other processes can then ask the driver to perform specific actions depending on the type of device, using normalized methods which can be sent to the driver using the ASK_DRIVER method. The driver receives these informations through the DRIVER_METHOD_REQUEST notification.

The driver is also in charge of translating the interrupts of a device as well as eventual events polled from its (mapped) memory to normalized notifications which can then be sent to processes that subscribed to them using the related normalized methods.

A note on performances

Although hardware devices' interrupts are notified to the driver through service socket notifications, the latency is still minimal as soon as the driver listens to the RECV_SOCK_MSG signal, which like all signals uses interrupts and so guarantees a very low latency.

Methods

0x01 ENUM_DEVICES

Enumerate connected devices.

It's also possible to only count the number of devices matching the provided criterias by providing a start index and end index of 0.

Required permissions:

• devices.enum

Arguments:

• Start index (4 bytes)
• End index (4 bytes)
• Pattern (277 bytes)

Answer:

• Number of found devices globally (4 bytes)
• Number of devices listed in this answer (4 bytes)
• DDD of each device (512 bytes * number of devices)
• 0x01 if some devices were masked due to insufficient permissions, 0x00 else (1 byte)

Errors:

• 0x10: Start index is lower than the end index
• 0x11: Invalid connection type
• 0x12: Bus number was provided without a connection type
• 0x13: Port number was provided without a connection type
• 0x14: Both bus number and port number were provided
• 0x15: Invalid DTD
• 0x20: Range is greater than the available answer size
• 0x21: Provided bus was not found
• 0x22: Provided port was not found

0x02 SUBSCRIBE_DEVICES

Subscribe to events related to devices matching a patterns.
All current and future devices matching this pattern will cause a DEVICE_EVENT notification.

Required permissions:

• devices.subscribe

Arguments:

• 0x00 to subscribe, any other value to unsubscribe
• Pattern (277 bytes)

Answer:

None

Errors:

• 0x20: Asked to unsubscribe but no subscription is active for this pattern

0x10 REGISTER_DRIVER

Set up a service as a driver for all devices matching a pattern.
If multiple drivers have colliding patterns, the final user will be prompted to choose a driver.

The driver process will receive DEVICE_EVENT notifications for drivable devices. This notification will only be sent for devices for which the system chose this driver as the main one.
Notifications are also retroactive, which means they will be sent for already-connected devices.

The driver will also have the device registered in its drivable devices attribute, allowing it to use the DEVICE_AMS syscall to map the device's memory in its own.

Required permissions:

• devices.register_driver

Arguments:

• Pattern of the devices to drive (277 bytes)

Answer:

None

Errors:

• 0x20: Current process is not a service
• 0x30: Current process is already registered as a driver for this pattern

0x11 UNREGISTER_DRIVER

Unregister a service previously registered as a driver.

Required permissions:

None

Arguments:

• Pattern to unsubscribe from (277 bytes)

Errors:

• 0x30: Current process is not registered as a driver for this pattern

0x20 NOTIFY_PROCESS

Send a notification to a process that registered itself for normalized methods through a normalized method.

Required permissions:

None

Arguments:

• Notification ID (8 bytes)
• Normalized notification's content

Answer:

Expected answer by the notified process for this method if any

Errors:

• 0x20: Unknown notification ID

0xA0 ASK_DRIVER

Ask a driver to use a normalized method on a device it drives.

Required permissions:

• devices.ask_driver

Arguments:

• Device's SDI (4 bytes)
• Method's code (4 bytes)
• Method's arguments (size depends on the method)

Answer:

Expected answer format for this method

Errors:

• 0x20: Unknown device SDI provided
• 0x21: Provided method code is invalid for this device
• 0x22: Invalid arguments provided for this method

Notifications

DEVICE_EVENT

Sent for a specific device to processes that either:

• Drive this specific device
• Subscribed to it using the SUBSCRIBE_DEVICES method

Datafield:

• DDD (512 bytes)
• Event code (1 byte):
  • 0x10: device was just connected
  • 0x11: a driver was just selected for the device
  • 0x12: the device is ready to use
  • 0x20: device was disconnected (software)
  • 0x21: the device is being disconnected by its driver
  • 0x22: the device has been disconnected by the driver
  • 0x23: the device was brutally disconnected (hardware)
  • 0x30: device was just put to sleep
  • 0x31: device was just awoken from sleep
• Indicator (1 byte):
  • Bit 0: set if this device is connected for the first time
  • Bit 1: set if this device was disconnected brutally (not by the system itself)
  • Bit 2: set if this device is connected for the first time on this specific port

DEVICE_INTERRUPT

Sent to a driver after a device it's currently driving raised an interrupt.

Datafield:

• Device's SDI (4 bytes)
• Normalized interrupt

DRIVER_METHOD_REQUEST

Sent to a driver after receiving a valid normalized method request.

The driver is expected to answer using the relevant answer format for the provided normalized method and arguments.

The notification ID is generated by this service to allow the driver to send normalized notifications to a process that registers for it through this method without showing the caller process' PID to the driver process.

Datafield:

• DDT (4 bytes)
• Notification ID (8 bytes)
• Method's code (4 bytes)
• Method's arguments (size depends on the method)

Expected answer:

Expected answer format for this method if any

DEVICE_NORM_NOTIF

Sent to a process that subscribed to normalized notifications of a device.
This notification is transferred by the sys::hw service after the driver sent it its content through the NOTIFY_PROCESS method.

Datafield:

• Device's SDI (4 bytes)
• Normalized notification's content

sys::perm service

The sys::perm service is used to manage the privileges of users and the permissions of processes.

The purpose of user privileges

In NightOS, executable instructions can run in three different contexts:

• In applications
• In system services
• In the kernel itself

The kernel doesn't have any limitation on what tasks it is allowed to perform, of course, as it is the one to decide.
System services communicate directly with the kernel and are trusted processes so they can do anything in their domain, which means for instance the sys::net cannot ask to manipulate the filesystem, as it's the role of sys::fs.

But applications, who run userland processes

List of permissions

TODO

Methods

TODO

Notifications

TODO

sys::net service

Methods

TODO

Notifications

TODO

sys::crypto service

Methods

TODO

Notifications

TODO

sys::crashsave service

Methods

TODO

Notifications

TODO

sys::flow service

Methods

TODO

Notifications

TODO

sys::hydre service

Methods

TODO

Notifications

TODO

Native applications

This folder contains documentation for each native application.

NOTE: Some documents may not be available at the moment.

Below is the list of applications that are installed (by default) during the installation process.

Userland interactive applications

Basics

• Nova : Desktop environment
• BareEnv : Text-based desktop environment
• Comet : File manager
• Stellar : Application store
• Rocket : Web browser
• Pluton : Terminal
• TimeTravel : Backup and versioning program

System utilities

• Central : Settings
• Monitor : Monitor opened applications, processes, CPU/RAM usage, etc.
• Registry : View and edit the registry (for advanced users)
• Skyer : Applications manager
• Cloudy : Backup & Sync manager

Utilities

• Gravity : Text editor
• Thinker : Notes and task lists manager
• ShootingStar : Pictures viewer and simple editor
• Sonata : Music player (with Hi-Res support and options)
• Milkshake : Video player
• Blackhole : Archives manager
• Reader : A complete e-book reader with many supported formats and options
• Postal : An e-mail client

Suites

• Particle : A complete IDE to make applications
• Astral : The complete toolchain allowing to build applications

Security

• Vortex : Firewall
• Locky : System's encryption tool, with the ability to manage encrypted archives and volumes

System services

System services are listed in their own specifications directory.

Astral

Astral is the toolchain required to build NightOS applications.

BareEnv

BareEnv, for bare environment, is text-based desktop environment meant for servers as well as fallback when no graphical output is available or when the primary desktop environment does not work for any reason.

Blackhole

Blackhole is the default archive manager of NightOS. It supports many different archives types (zip, 7z, gzip, etc.).

Central

The Central application delivers the Control Center which allows to configure how the system behaves. It is split into three distinct categories.

> Current user settings
    > User Account
        > Set profile picture
        > Set nickname

    > Change security level
        | Basic
        | Standard
        | Restricted
        | Extreme
        | Total
            ! Only available in developer mode

> Applications management
    > Sideloading
        > Change sideloading mode
            | Disabled
            | Secure
            | Unsecure

> Global settings
    > Date & Time
        > Set date
        > Set time
        > Change timezone
        > Synchronize time

    > Encryption
        > Manage storage encryption
            | Enable
            | Disable

        > Change encryption password
            ! Disabled if storage is not encrypted yet

        > Manage per-user encryption
            | Enable
            | Disable

> Users
    > Create new user
    > Create new administrator user
    > Create new supervised user

Development-related options

This part show settings that are only available in developer mode.

> Applications
    > Set application proxies
        | Enable
        | Disable

Cloudy

While the system and main data can be backed up using TimeTravel, it's not always handy to manage backups on an external hard drive and keep it safe.

This is where Cloudy comes: it allows to synchronize data on the cloud, using a master password which is stored nowhere but in the user's mind.

How it works

In order to work, Cloudy needs a storage account where it can store its data. Multiple providers are supported: Dropbox, Google Drive, Amazon AWS, SFTP storages, etc.

It will then store all data on this account, in a dedicated folder (the path can be customized). All data will be encrypted by a master password prompted when the application is opened for the first time, so no one will be able to access these date except the user who set it up itself.

Integration

Applications can ask to store and synchronize their own data through Cloudy. They will not be able to access any other application's data, though.

Collected data

The data that can be backup-ed and synchronized by Cloudy are:

• User's settings, computer's settings if the user is administrator
• Installed applications and their data
• Applications can ask to store additional custom data (e.g. a password manager)

Synchronization

The synchronization process has two parts:

• The up-sync process, which sends new data to the cloud ;
• The down-sync process, which reflects these changes on the local computer

Up-sync can be performed manually at anytime, or scheduled to be performed frequently. It's also possible to ask, for instance, to update the list of applications when a new one is installed or removed, while only backuping applications' data once a day.

Down-sync can be performed anytime, and a preview will show which items will be restored and what will be overridden. It's possible to only restore a few items. By default, down-sync will be performed in real-time: as soon as an up-sync process is performed, down-sync will happen on every other computer of the synchronization chain.

Synchronization chain

Data are synchronized through all computers of a single synchronization chain, which is simply a set of computers. Synchronization happens per user, which is why Cloudy doesn't backup every user of the computer, but only the one synchronization is enabled on. Of course, every user can enable synchronization for its account.

When a user enabled Cloudy for its account, it must connect with a NightOS account, and it is then added to the synchronization chain consisting of every computer having Cloudy enabled with this NightOS account.

For down-sync to happen, the username must be on the receiving computers than on the upload computer, else a specific setting will need to be changed in Cloudy.

Comet

Comet is the default file manager of NightOS.

Gravity

Gravity is a text editor featuring syntax highlighting for code.

Locky

Locky is NightOS' encryption tool. It is used to encrypt the whole storage and encrypt individual users' data.

Milkshake

Milkshake is the default video player of NightOS.

Monitor

Monitor allows to see and manage running applications and processes, as well as to watch resources usage like CPU percentage or used RAM.

Nova

Nova is the desktop environment of NightOS.

Particle

Particle is a complete IDE for making NightOS applications. It uses the Astral toolchain.

Pluton

Pluton is the default terminal of NightOS. It features a strong integration of Hydre.

Postal

Postal is the default e-mail client of NightOS.

Reader

Reader is an e-book reader with supports for comics. It can also manage libraries of books.

Registry

Registry allows to view and edit the registry. It is reserved to advanced users.

Rocket

Rocket is the default web browser of NightOS.

ShootingStar

ShootingStar is the default pictures viewer and editor of NightOS. Edition is limited to simple manipulations such as cropping and resizing.

Skyer

Skyer is the applications manager of NightOS. It allows to install and remove applications, as well as to manage their permissions and enable application proxies in developer mode.

Sonata

Sonata is the default music player of NightOS. It features support for Hi-Res audio files and can play many different formats (e.g. 32 bits / 384 kHz FLAC, DSD, ...). It also supports bit-to-bit playing to USB DACs.

Stellar

Thinker

Thinker is a simple notes application.

TimeTravel

TimeTravel is the backup and versioning program of NightOS.

Vortex

Vortex is NightOS' built-in firewall. It can also be used in combination with parental control.