FreeNAS/freenas 2cdf1fcsrc/middlewared/middlewared/plugins boot.py, src/middlewared/middlewared/plugins/disk_ swap_configure.py

NAS-128772 / 24.10 / cache boot disks (#13662)

* cache boot.get_disks

* update boot.get_disks cache on zfs event

* improve disk_/swap_configure.py
DeltaFile
+21-12src/middlewared/middlewared/plugins/disk_/swap_configure.py
+27-3src/middlewared/middlewared/plugins/boot.py
+6-2src/middlewared/middlewared/plugins/zfs_/zfs_events.py
+54-173 files

FreeNAS/freenas 265e80bsrc/middlewared/middlewared/plugins filesystem.py pwenc.py

NAS-128404 / 24.04.1 / Improve file receive (by yocalebo) (#13671)

* improve filesystem.file_receive

(cherry picked from commit e26d1d84c002aba1670eef488f46e006e7068334)

* fix imports

---------

Co-authored-by: caleb <yocalebo at gmail.com>
DeltaFile
+26-16src/middlewared/middlewared/plugins/filesystem.py
+2-1src/middlewared/middlewared/plugins/pwenc.py
+28-172 files

FreeNAS/freenas e48b017src/middlewared/middlewared/plugins filesystem.py

fix imports
DeltaFile
+2-2src/middlewared/middlewared/plugins/filesystem.py
+2-21 files

FreeNAS/freenas 2651167src/middlewared/middlewared/plugins filesystem.py pwenc.py

improve filesystem.file_receive

(cherry picked from commit e26d1d84c002aba1670eef488f46e006e7068334)
DeltaFile
+28-18src/middlewared/middlewared/plugins/filesystem.py
+2-1src/middlewared/middlewared/plugins/pwenc.py
+30-192 files

FreeNAS/freenas 6849183src/middlewared/middlewared/plugins filesystem.py pwenc.py

improve filesystem.file_receive (#13600)

DeltaFile
+26-17src/middlewared/middlewared/plugins/filesystem.py
+2-1src/middlewared/middlewared/plugins/pwenc.py
+28-182 files

FreeNAS/freenas 18820fcsrc/middlewared/middlewared/plugins/disk_ swap_configure.py

improve disk_/swap_configure.py
DeltaFile
+21-12src/middlewared/middlewared/plugins/disk_/swap_configure.py
+21-121 files

FreeNAS/freenas 15ae93csrc/middlewared/middlewared/plugins/zfs_ zfs_events.py

update boot.get_disks cache on zfs event
DeltaFile
+6-2src/middlewared/middlewared/plugins/zfs_/zfs_events.py
+6-21 files

FreeNAS/freenas 2e726d8src/middlewared/middlewared/plugins boot.py

cache boot.get_disks
DeltaFile
+27-3src/middlewared/middlewared/plugins/boot.py
+27-31 files

FreeNAS/freenas 27c6fa1src/middlewared/middlewared/plugins ldap.py directoryservices.py, src/middlewared/middlewared/plugins/smb_ util_smbconf.py passdb.py

Remove legacy samba domain support

DragonFish was last release where we support this legacy feature.
This commit removes most of middleware code, but leaves database
untouched. Next related PR for FreeIPA support will finalize
schema changes.
DeltaFile
+1-264tests/api2/test_275_ldap.py
+0-47src/middlewared/middlewared/plugins/ldap.py
+0-39src/middlewared/middlewared/plugins/smb_/util_smbconf.py
+1-6src/middlewared/middlewared/plugins/smb_/passdb.py
+0-5src/middlewared/middlewared/plugins/directoryservices.py
+0-3src/middlewared/middlewared/plugins/smb_/groupmap.py
+2-3641 files not shown
+2-3657 files

FreeNAS/freenas ff7b512tests/api2 test_261_iscsi_cmd.py test_300_nfs.py

Remove tests
DeltaFile
+0-2,741tests/api2/test_261_iscsi_cmd.py
+0-1,810tests/api2/test_300_nfs.py
+0-1,492tests/api2/test_200_ftp.py
+0-1,131tests/api2/test_341_pool_dataset_encryption.py
+0-946tests/api2/test_345_acl_nfs4.py
+0-818tests/api2/test_425_smb_protocol.py
+0-8,938215 files not shown
+0-35,234221 files

FreeNAS/freenas 2b802ectests/api2 test_275_ldap.py

Fix
DeltaFile
+0-37tests/api2/test_275_ldap.py
+0-371 files

FreeNAS/freenas fe1c985tests/api2 test_030_activedirectory.py

In test_07_enable_leave_activedirectory for HA test vip rather than ip
DeltaFile
+5-3tests/api2/test_030_activedirectory.py
+5-31 files

FreeNAS/freenas 26a131fsrc/middlewared/middlewared/plugins ldap.py directoryservices.py, src/middlewared/middlewared/plugins/smb_ util_smbconf.py passdb.py

Remove legacy samba domain support

DragonFish was last release where we support this legacy feature.
This commit removes most of middleware code, but leaves database
untouched. Next related PR for FreeIPA support will finalize
schema changes.
DeltaFile
+1-227tests/api2/test_275_ldap.py
+0-47src/middlewared/middlewared/plugins/ldap.py
+0-39src/middlewared/middlewared/plugins/smb_/util_smbconf.py
+1-6src/middlewared/middlewared/plugins/smb_/passdb.py
+0-5src/middlewared/middlewared/plugins/directoryservices.py
+0-3src/middlewared/middlewared/plugins/smb_/groupmap.py
+2-3271 files not shown
+2-3287 files

FreeNAS/freenas c5e7888src/middlewared/middlewared/plugins etc.py, src/middlewared/middlewared/pytest/unit/utils test_write_if_changed.py

NAS-128802 / 24.10 / Improve configuration file generation (#13669)

1. use os.rename to overwrite config files. Avoid possibility of
having applications read partially written configuration files by
first writing the changed config to a temporary file, then renaming
over existing file.

2. have write_if_changed allow specifying the desired permissions
and ownership of the file. This allows us to ensure that ownership and
permissions changes happen before we close and rename the file.

3. alter return value for write_if_changed from boolean indicating
whether changes were written to a bitmask of changes made. This allows
us to flag when permissions and ownership had to be altered (logging
potentially unexpected CLI changes).

4. add optional non-default parameter to raise an exception if
we had to change ownership and permissions. This exception can be
used in future to generate audit messages / alerts if we had

    [2 lines not shown]
DeltaFile
+191-0src/middlewared/middlewared/pytest/unit/utils/test_write_if_changed.py
+127-14src/middlewared/middlewared/utils/io.py
+20-30src/middlewared/middlewared/plugins/etc.py
+338-443 files

FreeNAS/freenas 1384ebbsrc/middlewared/middlewared/pytest/unit/utils test_write_if_changed.py

Expand tests
DeltaFile
+31-0src/middlewared/middlewared/pytest/unit/utils/test_write_if_changed.py
+31-01 files

FreeNAS/freenas 8273474src/middlewared/middlewared/utils io.py

Clarify `raise_error`
DeltaFile
+2-2src/middlewared/middlewared/utils/io.py
+2-21 files

FreeNAS/freenas 53b9443src/middlewared/middlewared/utils io.py

Add more validation
DeltaFile
+11-1src/middlewared/middlewared/utils/io.py
+11-11 files

FreeNAS/freenas 342bd4dsrc/middlewared/middlewared/utils io.py

Add documentation for dirfd
DeltaFile
+3-0src/middlewared/middlewared/utils/io.py
+3-01 files

FreeNAS/freenas 20f3aa1src/middlewared/middlewared/plugins etc.py

Stop raising exception
DeltaFile
+1-1src/middlewared/middlewared/plugins/etc.py
+1-11 files

FreeNAS/freenas 1100ad7src/middlewared/middlewared/plugins etc.py, src/middlewared/middlewared/pytest/unit/utils test_write_if_changed.py

Address review
DeltaFile
+8-7src/middlewared/middlewared/plugins/etc.py
+1-1src/middlewared/middlewared/pytest/unit/utils/test_write_if_changed.py
+1-1src/middlewared/middlewared/utils/io.py
+10-93 files

FreeNAS/freenas bd636a0src/middlewared/middlewared/plugins etc.py

Improve error messages
DeltaFile
+7-1src/middlewared/middlewared/plugins/etc.py
+7-11 files

FreeNAS/freenas 7eeb9d0src/middlewared/middlewared/plugins smb.py, src/middlewared/middlewared/plugins/smb_ util_param.py

Improve validation of SMB options (#13668)

Users will sometimes set completely invalid global SMB auxiliary
parameters in the smb plugin such as "oplocks = no # does this work".

This is rejected by libparam and breaks samba entirely if written
as-is. This commit improves validation by running global parameters
through loadparm_ctx.set() so that we can check both the parameter
name and its value.
DeltaFile
+14-9src/middlewared/middlewared/plugins/smb_/util_param.py
+5-6src/middlewared/middlewared/plugins/smb.py
+19-152 files

FreeNAS/freenas 641b29csrc/middlewared/middlewared/plugins boot.py, src/middlewared/middlewared/plugins/zfs_ zfs_events.py

update cache on zfs event
DeltaFile
+6-2src/middlewared/middlewared/plugins/zfs_/zfs_events.py
+0-3src/middlewared/middlewared/plugins/boot.py
+6-52 files

FreeNAS/freenas 0321983src/middlewared/middlewared/plugins/smb_ util_param.py

Fix
DeltaFile
+1-1src/middlewared/middlewared/plugins/smb_/util_param.py
+1-11 files

FreeNAS/freenas cfae219src/middlewared/middlewared/plugins/directoryservices_ cache.py

Fix AD cache retrieval issue (#13666)

Due to legacy behavior with dscache.get_uncached_user we passed
None for fields that were not used. user.get_user_obj and
group.get_group_obj do not allow null values for these fields.
DeltaFile
+14-6src/middlewared/middlewared/plugins/directoryservices_/cache.py
+14-61 files

FreeNAS/freenas 19f42c5src/middlewared/middlewared/etc_files/netdata netdata.conf.mako, src/middlewared/middlewared/plugins/reporting utils.py rest.py

Retrieve cgroup metrics selectively and update approximation logic
DeltaFile
+36-28src/middlewared/middlewared/pytest/unit/plugins/reporting/test_netdata_approximation.py
+35-1src/middlewared/middlewared/plugins/reporting/utils.py
+4-1src/middlewared/middlewared/etc_files/netdata/netdata.conf.mako
+3-0src/middlewared/middlewared/plugins/reporting/rest.py
+78-304 files

FreeNAS/freenas be1db30src/middlewared/middlewared/plugins etc.py, src/middlewared/middlewared/pytest/unit/utils test_write_if_changed.py

Improve configuration file generation

1. use os.rename to overwrite config files. Avoid possibility of
having applications read partially written configuration files by
first writing the changed config to a temporary file, then renaming
over existing file.

2. have write_if_changed allow specifying the desired permissions
and ownership of the file. This allows us to ensure that ownership and
permissions changes happen before we close and rename the file.

3. alter return value for write_if_changed from boolean indicating
whether changes were written to a bitmask of changes made. This allows
us to flag when permissions and ownership had to be altered (logging
potentially unexpected CLI changes).

4. add optional non-default parameter to raise an exception if
we had to change ownership and permissions. This exception can be
used in future to generate audit messages / alerts if we had

    [3 lines not shown]
DeltaFile
+161-0src/middlewared/middlewared/pytest/unit/utils/test_write_if_changed.py
+114-14src/middlewared/middlewared/utils/io.py
+13-30src/middlewared/middlewared/plugins/etc.py
+288-443 files

FreeNAS/freenas 056bec1src/middlewared/middlewared/pytest/unit/utils test_write_if_changed.py, src/middlewared/middlewared/utils io.py

Fix
DeltaFile
+1-1src/middlewared/middlewared/utils/io.py
+1-0src/middlewared/middlewared/pytest/unit/utils/test_write_if_changed.py
+2-12 files

FreeNAS/freenas 6e26695src/middlewared/middlewared/pytest/unit/utils test_write_if_changed.py

Fix
DeltaFile
+2-2src/middlewared/middlewared/pytest/unit/utils/test_write_if_changed.py
+2-21 files

FreeNAS/freenas 62a7419src/middlewared/middlewared/utils io.py

Fix
DeltaFile
+1-8src/middlewared/middlewared/utils/io.py
+1-81 files