Add customized memory and cpu for function

https://storyboard.openstack.org/#!/story/2001586

Only users know the amount of resources that their functions may need, so qinling should provide cpu and memory_size options for users to customize resources. When creating function, allow user to specify cpu and memory_size resources for the function so that qinling can allocate and manage resources of cpu and memory more reasonably in this way.

Problem description

  • For deployers, they may want to manage system resources reasonably and safely by restricting functions’ resource occupancy so that more executions could share the fixed amount of resources and non-interfering with each other.

  • For users, applying for resources according to actual needs can help them prevent the waste of resources, which will make sense when users need to pay for the resources they use.

Proposed change

For different types of functions, the ways of limiting resources are distinguishing.

  • For image type functions, we could config the resource limitation in pod definition, and then kubernetes will help us limit the whole pod, which means the total amount of resources used by the function inside the pod could never exceed the pod’s limit.

  • For package/swift type function, it turns to be the runtime server’s responsibility to set resource limits for subprocess that function runs in.

Some details are as following:

First of all, make sure the values of memory_size and cpu saved in the function database are valid. In api.controllers.v1.function.post, we do type and size check for cpu and memory_size params so that they must be integers and within the range set in config.py, and qinling will supply default values for them if users do not input anything. The default values are set to be the minimum in this range.

Besides, in api.controllers.v1.function.put, we add cpu and memory_size to UPDATE_ALLOWED. If user wants to update them, we will do type and size check for cpu and memory_size params before updating function database.

When creating execution, pass cpu and memory_size values saved in function database to qinling.engine.default_engine.create_execution. Here we will do the check for cpu and memory_size values again in case the limited scope of resources set in config.py have been reset.

Then for image type function, pass both of them to orchestrator.kubernetes.manager._create_pod, and we can get a pod with limited cpu and memory resources. For package/swift type function, pass both of them to the worker pod in the k8s deployment by using qinling.engine.utils.url_request. In the selected worker pod, cpu and memory_size values will be used to limit the total amount of resources that function process and its subprocesses can use.

We are considering using ‘cgroup’ in linux to limit cpu and memory because by ‘cgroup’ we can add pids of function process and its subprocesses to the same ‘control group’ and limit the total amount of resources. But we need to use different users to set resource files in ‘cgroup’ and run functions because function should not be granted permission to modify the resource setting files in ‘cgroup’. We use root to create a ‘control group’, and qinling can only write to the ‘tasks’ file.

  • For cpu resource limitation, ‘cpu.cfs_quota_us’ and ‘cpu.cfs_period_us’ files in ‘cgroup’ will be used to convert millicpu value.

  • For memory resource limitation, only ‘memory.limit_in_bytes’ file will be used to limit RAM because now in k8s source code, the kubelet does not support running with swap enabled. Although it also provides a workaround ‘–fail-swap-on=false’ to allow swap on, which may cause some performance impacts, we would better to disable swap now. For more details about swap, please see the references.

Data model impact

Add a cpu column for function database to save cpu value specified by user.

REST API impact

  • Add cpu and memory_size options for function creation.

  • Allow to update cpu and memory_size values saved in function database.

End user impact

When using python-qinlingclient to create/update function, the CLI may look like ‘openstack function create/update –cpu xxx –memory_size xxx …’.

Performance Impact

None.

Deployer impact

The config options for min/max size of cpu and memory_size will be provided in qinling.config. The unit of cpu is ‘millicpu’ and the unit of memory_size is ‘bytes’.

Alternatives

We have considered using resource.RLIMIT_AS to limit memory resource that function can use. However if function forks other child processes, the child processes will inherits its parent’s resource limits, instead of sharing the limits.

Implementation

Assignee(s)

Jiangyuan <yuan.jiang@easystack.cn>

Dependencies

None.

Testing

None.