[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:13 dist-test-slave-zl47 krb5kdc[2927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:13 dist-test-slave-zl47 krb5kdc[2927](info): set up 2 sockets
Sep 23 14:36:13 dist-test-slave-zl47 krb5kdc[2927](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 14:36:14.798245 2920 mini_kdc.cc:121] Time spent starting KDC: real 1.430s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:14 dist-test-slave-zl47 krb5kdc[2927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:14.902328 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36345
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:36345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:14.993283 2940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:14.993510 2940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:14.993575 2940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:14.997198 2940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:14.997278 2940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:14.997314 2940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:14.997364 2940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:14.997397 2940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:15.001758 2940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:36345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:15.002614 2940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:15.003520 2940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:15.010921 2948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:15.011099 2946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:15.011024 2945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:15.010972 2940 server_base.cc:1034] running on GCE node
I20260923 14:36:15.011793 2940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:15.012794 2940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:15.013950 2940 hybrid_clock.cc:648] HybridClock initialized: now 1790174175013934 us; error 26 us; skew 500 ppm
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:15.016384 2940 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:15.017061 2940 webserver.cc:460] Webserver started at http://127.2.218.62:37835/ using document root <none> and password file <none>
I20260923 14:36:15.017511 2940 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:15.017570 2940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:15.017759 2940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:15.019413 2940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "c691e7ac69464eb28c9d600fcd263e54"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "aecc28ce01242969ba71bb71ec90479c"
server_key_iv: "3567217f6cfe5469037d9131fcff12b0"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.019901 2940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "c691e7ac69464eb28c9d600fcd263e54"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "aecc28ce01242969ba71bb71ec90479c"
server_key_iv: "3567217f6cfe5469037d9131fcff12b0"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.023087 2940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260923 14:36:15.025063 2955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:15.026046 2940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:15.026178 2940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "c691e7ac69464eb28c9d600fcd263e54"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "aecc28ce01242969ba71bb71ec90479c"
server_key_iv: "3567217f6cfe5469037d9131fcff12b0"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.026273 2940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:15.033751 2940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:15.035989 2940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:15.036144 2940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:15.041954 2940 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:36345
I20260923 14:36:15.041970 3007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:36345 every 8 connection(s)
I20260923 14:36:15.042704 2940 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:15.045109 3008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:15.046514 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 2940
I20260923 14:36:15.046693 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:15.048017 2920 external_mini_cluster.cc:1428] Setting key 84e602e42b0e0343905b915bc6ba6db6
I20260923 14:36:15.050706 3008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54: Bootstrap starting.
I20260923 14:36:15.052749 3008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:15.053398 3008 log.cc:824] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:15.054996 3008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54: No bootstrap required, opened a new log
I20260923 14:36:15.058270 3008 raft_consensus.cc:348] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c691e7ac69464eb28c9d600fcd263e54" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36345 } }
I20260923 14:36:15.058472 3008 raft_consensus.cc:374] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:15.058514 3008 raft_consensus.cc:729] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c691e7ac69464eb28c9d600fcd263e54, State: Initialized, Role: FOLLOWER
I20260923 14:36:15.058879 3008 consensus_queue.cc:260] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c691e7ac69464eb28c9d600fcd263e54" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36345 } }
I20260923 14:36:15.058985 3008 raft_consensus.cc:388] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:15.059060 3008 raft_consensus.cc:482] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:15.059168 3008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:15.059849 3008 raft_consensus.cc:504] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c691e7ac69464eb28c9d600fcd263e54" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36345 } }
I20260923 14:36:15.060146 3008 leader_election.cc:302] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c691e7ac69464eb28c9d600fcd263e54; no voters:
I20260923 14:36:15.060398 3008 leader_election.cc:288] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:15.060491 3013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:15.060684 3013 raft_consensus.cc:686] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [term 1 LEADER]: Becoming Leader. State: Replica: c691e7ac69464eb28c9d600fcd263e54, State: Running, Role: LEADER
I20260923 14:36:15.060971 3013 consensus_queue.cc:237] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c691e7ac69464eb28c9d600fcd263e54" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36345 } }
I20260923 14:36:15.061487 3008 sys_catalog.cc:564] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:15.062333 3014 sys_catalog.cc:455] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c691e7ac69464eb28c9d600fcd263e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c691e7ac69464eb28c9d600fcd263e54" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36345 } } }
I20260923 14:36:15.062439 3014 sys_catalog.cc:458] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:15.062778 3022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:15.062832 3015 sys_catalog.cc:455] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c691e7ac69464eb28c9d600fcd263e54. Latest consensus state: current_term: 1 leader_uuid: "c691e7ac69464eb28c9d600fcd263e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c691e7ac69464eb28c9d600fcd263e54" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36345 } } }
I20260923 14:36:15.062995 3015 sys_catalog.cc:458] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54 [sys.catalog]: This master's current role is: LEADER
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:15.065645 3022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:15.069373 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:15.050970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.2.218.62:36345)
0923 14:36:15.051363 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:15.051377 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:15.052320 (+ 943us) server_negotiation.cc:407] Connection header received
0923 14:36:15.055097 (+ 2777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:15.055115 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:15.055423 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:15.055709 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:15.057326 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.058683 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.059823 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.060410 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.061870 (+ 1460us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:15.064757 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:15.064762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:15.064774 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:15.064798 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:15.066513 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.067027 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.067032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.067037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.067158 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.067508 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.067511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.067512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.067893 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:15.068056 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:15.068596 (+ 540us) server_negotiation.cc:299] Negotiation successful
0923 14:36:15.068829 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":112,"threads_started":1}
I20260923 14:36:15.071281 3022 catalog_manager.cc:1347] Generated new cluster ID: 7b5c5751f6594603bf37d30b29c8056e
I20260923 14:36:15.071333 3022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:15.083927 3022 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:15.084824 3022 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:15.088758 3022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c691e7ac69464eb28c9d600fcd263e54: Generated new TSK 0
I20260923 14:36:15.089248 3022 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:36:15.112916 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36345
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:15.207504 3034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:15.207700 3034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:15.207768 3034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:15.211145 3034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:15.211193 3034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:15.211295 3034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:15.215472 3034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36345
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:15.216248 3034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:15.217094 3034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:15.225430 3040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:15.225481 3034 server_base.cc:1034] running on GCE node
W20260923 14:36:15.225425 3042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:15.225456 3039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:15.225941 3034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:15.226468 3034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:15.227624 3034 hybrid_clock.cc:648] HybridClock initialized: now 1790174175227604 us; error 35 us; skew 500 ppm
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:15.229856 3034 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:36:15.230554 3034 webserver.cc:460] Webserver started at http://127.2.218.1:33527/ using document root <none> and password file <none>
I20260923 14:36:15.230985 3034 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:15.231065 3034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:15.231240 3034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:15.232764 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "243dc96e1f00488e99c9e848200058d1"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "4ff924c17946e74806cc3a54fe4b9608"
server_key_iv: "0f8be2adab9af27aeefac122e101a505"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.233237 3034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "243dc96e1f00488e99c9e848200058d1"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "4ff924c17946e74806cc3a54fe4b9608"
server_key_iv: "0f8be2adab9af27aeefac122e101a505"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.236439 3034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:36:15.238461 3049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:15.239363 3034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:15.239470 3034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "243dc96e1f00488e99c9e848200058d1"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "4ff924c17946e74806cc3a54fe4b9608"
server_key_iv: "0f8be2adab9af27aeefac122e101a505"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.239554 3034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:15.258841 3034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:15.261081 3034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:15.261232 3034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:15.261691 3034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:15.262377 3034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:15.262434 3034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:15.262485 3034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:15.262526 3034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:15.269913 3034 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:43893
I20260923 14:36:15.269943 3162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:43893 every 8 connection(s)
I20260923 14:36:15.270619 3034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:15.277765 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 3034
I20260923 14:36:15.277842 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:15.278067 2920 external_mini_cluster.cc:1428] Setting key 65d30eeb536ccd622ce6107ed461bc22
I20260923 14:36:15.281877 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:15.271957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:57889 (local address 127.2.218.62:36345)
0923 14:36:15.272062 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:15.272065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:15.272660 (+ 595us) server_negotiation.cc:407] Connection header received
0923 14:36:15.273742 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:15.273744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:15.273789 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:15.273859 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:15.274926 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.275389 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.276278 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.276479 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.276545 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:15.278577 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:15.278581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:15.278583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:15.278614 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:15.279795 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.280325 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.280328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.280329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.280393 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.280742 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.280744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.280745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.281053 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:15.281168 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:15.281529 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 14:36:15.281665 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:36:15.282410 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:15.272174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36345 (local address 127.2.218.1:57889)
0923 14:36:15.272529 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:15.272563 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:15.273535 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:15.273995 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:15.274000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:15.274371 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:15.274743 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:15.274765 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.275482 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.275484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:15.276188 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:15.276196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.276580 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.276582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:15.276693 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:15.277163 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:15.277176 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:15.278421 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:15.279897 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:15.279904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:15.279918 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:15.280243 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:15.280465 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:15.280471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:15.280475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:15.280632 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:15.281160 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:15.281163 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:15.281328 (+ 165us) client_negotiation.cc:769] Sending connection context
0923 14:36:15.281452 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 14:36:15.281633 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
I20260923 14:36:15.283599 3163 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36345
I20260923 14:36:15.283803 3163 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:15.284168 3163 heartbeater.cc:511] Master 127.2.218.62:36345 requested a full tablet report, sending...
I20260923 14:36:15.285392 2972 ts_manager.cc:194] Registered new tserver with Master: 243dc96e1f00488e99c9e848200058d1 (127.2.218.1:43893)
I20260923 14:36:15.286408 2972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:57889
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:36:15.301679 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36345
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:15.396766 3168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:15.396974 3168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:15.397040 3168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:15.400385 3168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:15.400455 3168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:15.400538 3168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:15.404726 3168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36345
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:15.405484 3168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:15.406271 3168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:15.413882 3174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:15.413889 3176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:15.413889 3173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:15.414007 3168 server_base.cc:1034] running on GCE node
I20260923 14:36:15.414448 3168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:15.414903 3168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:15.416066 3168 hybrid_clock.cc:648] HybridClock initialized: now 1790174175416027 us; error 50 us; skew 500 ppm
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:15.418358 3168 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:36:15.419059 3168 webserver.cc:460] Webserver started at http://127.2.218.2:34263/ using document root <none> and password file <none>
I20260923 14:36:15.419495 3168 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:15.419569 3168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:15.419760 3168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:15.421401 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "8b3a2022f4f4435bac38d273ee0ba933"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "8ac66a6824be9179e47aa139520ccadf"
server_key_iv: "6da368e2a6f970af14f591fdd927780f"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.421893 3168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "8b3a2022f4f4435bac38d273ee0ba933"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "8ac66a6824be9179e47aa139520ccadf"
server_key_iv: "6da368e2a6f970af14f591fdd927780f"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.424911 3168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260923 14:36:15.426913 3183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:15.427934 3168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:15.428054 3168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "8b3a2022f4f4435bac38d273ee0ba933"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "8ac66a6824be9179e47aa139520ccadf"
server_key_iv: "6da368e2a6f970af14f591fdd927780f"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.428148 3168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:15.468514 3168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:15.470865 3168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:15.471022 3168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:15.471467 3168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:15.472098 3168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:15.472153 3168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:15.472206 3168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:15.472245 3168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:15.479579 3168 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:45837
I20260923 14:36:15.479641 3296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:45837 every 8 connection(s)
I20260923 14:36:15.480305 3168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:36:15.486241 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 3168
I20260923 14:36:15.486351 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:15.486596 2920 external_mini_cluster.cc:1428] Setting key a0ec40420e94bb53ce508b137826e0f5
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:15.492333 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:15.481727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:57069 (local address 127.2.218.62:36345)
0923 14:36:15.481853 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:15.481874 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:15.482556 (+ 682us) server_negotiation.cc:407] Connection header received
0923 14:36:15.483770 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:15.483773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:15.483821 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:15.483893 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:15.484841 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.485292 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.486411 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.486639 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.486735 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:15.489074 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:15.489080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:15.489086 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:15.489117 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:15.490464 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.490994 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.490996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.490998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.491087 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.491432 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.491434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.491435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.491579 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:15.491646 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:15.492154 (+ 508us) server_negotiation.cc:299] Negotiation successful
0923 14:36:15.492233 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:36:15.493314 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:15.481981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36345 (local address 127.2.218.2:57069)
0923 14:36:15.482407 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:15.482442 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:15.483574 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:15.484004 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:15.484010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:15.484306 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:15.484732 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:15.484743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.485378 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.485380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:15.486294 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:15.486304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.486763 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.486766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:15.486945 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:15.487467 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:15.487480 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:15.488929 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:15.490558 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:15.490565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:15.490579 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:15.490906 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:15.491173 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:15.491177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:15.491180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:15.491344 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:15.491672 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:15.491678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:15.491939 (+ 261us) client_negotiation.cc:769] Sending connection context
0923 14:36:15.492140 (+ 201us) client_negotiation.cc:240] Negotiation successful
0923 14:36:15.492435 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":156,"threads_started":1}
I20260923 14:36:15.494277 3297 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36345
I20260923 14:36:15.494501 3297 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
I20260923 14:36:15.494937 3297 heartbeater.cc:511] Master 127.2.218.62:36345 requested a full tablet report, sending...
I20260923 14:36:15.495842 2972 ts_manager.cc:194] Registered new tserver with Master: 8b3a2022f4f4435bac38d273ee0ba933 (127.2.218.2:45837)
I20260923 14:36:15.496335 2972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:57069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:36:15.508800 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36345
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:15.637751 3302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:15.638037 3302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:15.638141 3302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:15.646229 3302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:15.646292 3302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:15.646411 3302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:15.656172 3302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36345
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:15.657146 3302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:15.658270 3302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:15.667680 3307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:15.667744 3310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:15.667753 3308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:15.668224 3302 server_base.cc:1034] running on GCE node
I20260923 14:36:15.668550 3302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:15.669066 3302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:15.670230 3302 hybrid_clock.cc:648] HybridClock initialized: now 1790174175670209 us; error 41 us; skew 500 ppm
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:15.672873 3302 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:36:15.673834 3302 webserver.cc:460] Webserver started at http://127.2.218.3:34125/ using document root <none> and password file <none>
I20260923 14:36:15.674445 3302 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:15.674520 3302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:15.674728 3302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:15.677081 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "a002a4961c3d4a10b72739e566cdf11a"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "d73db8379a9f8f213b74637981786253"
server_key_iv: "cf22117db74f69f46e4b2aec89533415"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.677706 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "a002a4961c3d4a10b72739e566cdf11a"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "d73db8379a9f8f213b74637981786253"
server_key_iv: "cf22117db74f69f46e4b2aec89533415"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.681855 3302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260923 14:36:15.684739 3317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:15.685988 3302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:36:15.686113 3302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "a002a4961c3d4a10b72739e566cdf11a"
format_stamp: "Formatted at 2026-09-23 14:36:15 on dist-test-slave-zl47"
server_key: "d73db8379a9f8f213b74637981786253"
server_key_iv: "cf22117db74f69f46e4b2aec89533415"
server_key_version: "encryptionkey@0"
I20260923 14:36:15.686220 3302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:15.705426 3302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:15.708806 3302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:15.708974 3302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:15.709571 3302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:15.710563 3302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:15.710631 3302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:15.710696 3302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:15.710731 3302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:15.721547 3302 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:41807
I20260923 14:36:15.721566 3430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:41807 every 8 connection(s)
I20260923 14:36:15.722553 3302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:36:15.723593 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 3302
I20260923 14:36:15.723708 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:15.723961 2920 external_mini_cluster.cc:1428] Setting key fd17921db0b5a50b115e4953ab524879
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:15.735646 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:15.724179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:48071 (local address 127.2.218.62:36345)
0923 14:36:15.724299 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:15.724304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:15.725245 (+ 941us) server_negotiation.cc:407] Connection header received
0923 14:36:15.726529 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:15.726532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:15.726585 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:15.726663 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:15.727594 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.728071 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.729059 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.729314 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.729412 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:15.732154 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:15.732157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:15.732159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:15.732180 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:15.733481 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.734091 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.734129 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.734133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.734221 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.734622 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.734624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.734626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.734791 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:15.734868 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:15.735431 (+ 563us) server_negotiation.cc:299] Negotiation successful
0923 14:36:15.735527 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:36:15.736616 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:15.724552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36345 (local address 127.2.218.3:48071)
0923 14:36:15.725117 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:15.725160 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:15.726336 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:15.726805 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:15.726811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:15.727098 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:15.727461 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:15.727472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.728187 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.728190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:15.728931 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:15.728939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.729454 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.729459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:15.729636 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:15.730227 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:15.730249 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:15.731997 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:15.733614 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:15.733622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:15.733637 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:15.733989 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:15.734308 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:15.734313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:15.734316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:15.734499 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:15.734884 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:15.734891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:15.735154 (+ 263us) client_negotiation.cc:769] Sending connection context
0923 14:36:15.735366 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 14:36:15.735692 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":117,"threads_started":1}
I20260923 14:36:15.737828 3431 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36345
I20260923 14:36:15.738039 3431 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:15.741384 3431 heartbeater.cc:511] Master 127.2.218.62:36345 requested a full tablet report, sending...
I20260923 14:36:15.742427 2972 ts_manager.cc:194] Registered new tserver with Master: a002a4961c3d4a10b72739e566cdf11a (127.2.218.3:41807)
I20260923 14:36:15.743243 2972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:48071
I20260923 14:36:15.749521 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:36:15.772184 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:15.764132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.2.218.1:43893)
0923 14:36:15.764394 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:15.764400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:15.764422 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:36:15.764473 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:15.764494 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:15.764644 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:15.764769 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:15.765370 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.766088 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.766824 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.767132 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.767232 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:15.769210 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:15.769218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:15.769227 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:15.769246 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:15.770640 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.771039 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.771044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.771045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.771116 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.771377 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.771394 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.771395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.771600 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:15.771769 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:15.771916 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 14:36:15.772037 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
W20260923 14:36:15.772894 3077 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50368
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[2927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:36:15.793700 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:15.786729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.2.218.1:43893)
0923 14:36:15.786838 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:15.786841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:15.786952 (+ 111us) server_negotiation.cc:407] Connection header received
0923 14:36:15.787073 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:15.787076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:15.787111 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:15.787176 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:15.787733 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.788176 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.788880 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:15.789086 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:15.789177 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:15.791114 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:15.791119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:15.791121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:15.791143 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:15.792549 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.792885 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.792887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.792888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.792953 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:15.793172 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:15.793173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:15.793174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:15.793321 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:15.793401 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:15.793527 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 14:36:15.793610 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:36:15.794584 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 3034
I20260923 14:36:15.799968 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 3168
I20260923 14:36:15.806623 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 3302
I20260923 14:36:15.811362 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 2940
2026-09-23T14:36:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2472 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[3445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[3445](info): set up 2 sockets
Sep 23 14:36:15 dist-test-slave-zl47 krb5kdc[3445](info): commencing operation
krb5kdc: starting...
W20260923 14:36:16.709939 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.877s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:16 dist-test-slave-zl47 krb5kdc[3445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:16.809965 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:43105
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:16.903618 3458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:16.903848 3458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:16.903911 3458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:16.907539 3458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:16.907616 3458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:16.907656 3458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:16.907720 3458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:16.907763 3458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:16.912331 3458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:43105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:16.913122 3458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:16.913916 3458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:16.920928 3466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:16.921079 3463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:16.921059 3464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:16.920984 3458 server_base.cc:1034] running on GCE node
I20260923 14:36:16.921634 3458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:16.922583 3458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:16.923733 3458 hybrid_clock.cc:648] HybridClock initialized: now 1790174176923711 us; error 31 us; skew 500 ppm
Sep 23 14:36:16 dist-test-slave-zl47 krb5kdc[3445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174176, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:16.925987 3458 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:16.926721 3458 webserver.cc:460] Webserver started at http://127.2.218.62:43527/ using document root <none> and password file <none>
I20260923 14:36:16.927175 3458 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:16.927246 3458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:16.927438 3458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:16.929080 3458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "58ba41b60c024de3937552dbb06a86f5"
format_stamp: "Formatted at 2026-09-23 14:36:16 on dist-test-slave-zl47"
server_key: "5c65db3d13e0d5785f2f4d38b35ffe2f"
server_key_iv: "606ddf365cf4fa61e01723832244ed0a"
server_key_version: "encryptionkey@0"
I20260923 14:36:16.929592 3458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "58ba41b60c024de3937552dbb06a86f5"
format_stamp: "Formatted at 2026-09-23 14:36:16 on dist-test-slave-zl47"
server_key: "5c65db3d13e0d5785f2f4d38b35ffe2f"
server_key_iv: "606ddf365cf4fa61e01723832244ed0a"
server_key_version: "encryptionkey@0"
I20260923 14:36:16.932659 3458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260923 14:36:16.934648 3473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:16.935632 3458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:36:16.935757 3458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "58ba41b60c024de3937552dbb06a86f5"
format_stamp: "Formatted at 2026-09-23 14:36:16 on dist-test-slave-zl47"
server_key: "5c65db3d13e0d5785f2f4d38b35ffe2f"
server_key_iv: "606ddf365cf4fa61e01723832244ed0a"
server_key_version: "encryptionkey@0"
I20260923 14:36:16.935854 3458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:16.954137 3458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:16.956638 3458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:16.956812 3458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:16.963222 3458 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:43105
I20260923 14:36:16.963230 3525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:43105 every 8 connection(s)
I20260923 14:36:16.963986 3458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:16.966229 3526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:16.970669 3526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5: Bootstrap starting.
I20260923 14:36:16.972631 3526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:16.973315 3526 log.cc:824] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:16.974157 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 3458
I20260923 14:36:16.974270 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:16.974504 2920 external_mini_cluster.cc:1428] Setting key 764ff11739caff52750567129975d405
I20260923 14:36:16.975234 3526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5: No bootstrap required, opened a new log
I20260923 14:36:16.978518 3526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ba41b60c024de3937552dbb06a86f5" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43105 } }
I20260923 14:36:16.978720 3526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:16.978787 3526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58ba41b60c024de3937552dbb06a86f5, State: Initialized, Role: FOLLOWER
I20260923 14:36:16.979187 3526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ba41b60c024de3937552dbb06a86f5" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43105 } }
I20260923 14:36:16.979311 3526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:16.979370 3526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:16.979452 3526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:16.980159 3526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ba41b60c024de3937552dbb06a86f5" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43105 } }
I20260923 14:36:16.980496 3526 leader_election.cc:302] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58ba41b60c024de3937552dbb06a86f5; no voters:
I20260923 14:36:16.980798 3526 leader_election.cc:288] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:16.980851 3532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:16.981040 3532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [term 1 LEADER]: Becoming Leader. State: Replica: 58ba41b60c024de3937552dbb06a86f5, State: Running, Role: LEADER
I20260923 14:36:16.981365 3532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ba41b60c024de3937552dbb06a86f5" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43105 } }
Sep 23 14:36:16 dist-test-slave-zl47 krb5kdc[3445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:16.981848 3526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:16.982683 3533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58ba41b60c024de3937552dbb06a86f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ba41b60c024de3937552dbb06a86f5" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43105 } } }
I20260923 14:36:16.983050 3533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:16.983474 3533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58ba41b60c024de3937552dbb06a86f5. Latest consensus state: current_term: 1 leader_uuid: "58ba41b60c024de3937552dbb06a86f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58ba41b60c024de3937552dbb06a86f5" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43105 } } }
I20260923 14:36:16.983558 3533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5 [sys.catalog]: This master's current role is: LEADER
W20260923 14:36:16.986392 3547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:36:16.986460 3547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:36:16.986524 3541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:16.986727 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:16.975405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.2.218.62:43105)
0923 14:36:16.975788 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:16.975800 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:16.975825 (+ 25us) server_negotiation.cc:407] Connection header received
0923 14:36:16.976340 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:16.976354 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:16.976609 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:16.976881 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:16.977523 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:16.978439 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:16.979315 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:16.979788 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:16.979953 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:16.982453 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:16.982459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:16.982472 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:16.982500 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:16.984287 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:16.984652 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:16.984656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:16.984661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:16.984780 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:16.985007 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:16.985010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:16.985011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:16.985409 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:16.985588 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:16.985816 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 14:36:16.986041 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":101,"threads_started":1}
I20260923 14:36:16.987219 3541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:16.992023 3541 catalog_manager.cc:1347] Generated new cluster ID: 31902e490b7747a0981a9bf6ffb92ef1
I20260923 14:36:16.992081 3541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:16.995935 3541 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:16.996913 3541 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:17.002233 3541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58ba41b60c024de3937552dbb06a86f5: Generated new TSK 0
I20260923 14:36:17.002813 3541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:36:17.035096 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:43105
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:17.140671 3553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:17.140872 3553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:17.140908 3553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:17.144302 3553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:17.144346 3553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:17.144393 3553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:17.148705 3553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:43105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:17.149469 3553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:17.150291 3553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:17.157959 3561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:17.158010 3559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:17.158136 3553 server_base.cc:1034] running on GCE node
W20260923 14:36:17.158034 3558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:17.158540 3553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:17.159039 3553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:17.160233 3553 hybrid_clock.cc:648] HybridClock initialized: now 1790174177160207 us; error 37 us; skew 500 ppm
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174177, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:17.162683 3553 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:36:17.163394 3553 webserver.cc:460] Webserver started at http://127.2.218.1:40069/ using document root <none> and password file <none>
I20260923 14:36:17.163856 3553 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:17.163933 3553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:17.164152 3553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:17.165871 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "9dcc45ed27694f17b07bf823587d5b45"
format_stamp: "Formatted at 2026-09-23 14:36:17 on dist-test-slave-zl47"
server_key: "25f4e2e8a581aa25221d441a2666b57e"
server_key_iv: "521982b47858a0efd12cc32c84cd803f"
server_key_version: "encryptionkey@0"
I20260923 14:36:17.166388 3553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "9dcc45ed27694f17b07bf823587d5b45"
format_stamp: "Formatted at 2026-09-23 14:36:17 on dist-test-slave-zl47"
server_key: "25f4e2e8a581aa25221d441a2666b57e"
server_key_iv: "521982b47858a0efd12cc32c84cd803f"
server_key_version: "encryptionkey@0"
I20260923 14:36:17.169456 3553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260923 14:36:17.171725 3568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:17.172890 3553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260923 14:36:17.173017 3553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "9dcc45ed27694f17b07bf823587d5b45"
format_stamp: "Formatted at 2026-09-23 14:36:17 on dist-test-slave-zl47"
server_key: "25f4e2e8a581aa25221d441a2666b57e"
server_key_iv: "521982b47858a0efd12cc32c84cd803f"
server_key_version: "encryptionkey@0"
I20260923 14:36:17.173159 3553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:17.203945 3553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:17.206125 3553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:17.206277 3553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:17.206730 3553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:17.207365 3553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:17.207420 3553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:17.207472 3553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:17.207506 3553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:17.214710 3553 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:35737
I20260923 14:36:17.214766 3683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:35737 every 8 connection(s)
I20260923 14:36:17.215397 3553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:17.220194 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 3553
I20260923 14:36:17.220288 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:17.220503 2920 external_mini_cluster.cc:1428] Setting key 0fdec8c28fab800f08376e300c4c9f54
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174177, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:17.227111 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.216780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:57579 (local address 127.2.218.62:43105)
0923 14:36:17.216894 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.216897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.217523 (+ 626us) server_negotiation.cc:407] Connection header received
0923 14:36:17.218774 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.218778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.218833 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.218908 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:17.219823 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.220485 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.221570 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.221841 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.221935 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.223868 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:17.223871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:17.223873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:17.223898 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:17.225192 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.225657 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.225663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.225667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.225748 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.226039 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.226044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.226048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.226313 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:17.226417 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.226784 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.226935 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:36:17.227528 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.217054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:43105 (local address 127.2.218.1:57579)
0923 14:36:17.217404 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:17.217438 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:17.218580 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:17.219075 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:17.219081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:17.219345 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:17.219700 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.219710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.220620 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.220623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.221446 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.221455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.221963 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.221966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.222141 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.222586 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:17.222599 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:17.223733 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:17.225312 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:17.225317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:17.225328 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:17.225549 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:17.225837 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:17.225839 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:17.225840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:17.225950 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:17.226408 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:17.226412 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:17.226585 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 14:36:17.226710 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 14:36:17.226891 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":96,"threads_started":1}
I20260923 14:36:17.228519 3684 heartbeater.cc:347] Connected to a master server at 127.2.218.62:43105
I20260923 14:36:17.228747 3684 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
I20260923 14:36:17.229206 3684 heartbeater.cc:511] Master 127.2.218.62:43105 requested a full tablet report, sending...
I20260923 14:36:17.230459 3490 ts_manager.cc:194] Registered new tserver with Master: 9dcc45ed27694f17b07bf823587d5b45 (127.2.218.1:35737)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
I20260923 14:36:17.231495 3490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:57579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:36:17.242820 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:43105
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:17.337142 3689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:17.337370 3689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:17.337442 3689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:17.341424 3689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:17.341519 3689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:17.341626 3689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:17.347599 3689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:43105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:17.348451 3689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:17.349326 3689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:17.357941 3697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:17.357964 3695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:17.358013 3689 server_base.cc:1034] running on GCE node
W20260923 14:36:17.357964 3694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:17.358592 3689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:17.359128 3689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:17.360309 3689 hybrid_clock.cc:648] HybridClock initialized: now 1790174177360297 us; error 23 us; skew 500 ppm
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174177, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:17.362833 3689 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:36:17.363633 3689 webserver.cc:460] Webserver started at http://127.2.218.2:39745/ using document root <none> and password file <none>
I20260923 14:36:17.364114 3689 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:17.364197 3689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:17.364394 3689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:17.366245 3689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "13d7e8ff8c9948d6b78480fd747cc6f1"
format_stamp: "Formatted at 2026-09-23 14:36:17 on dist-test-slave-zl47"
server_key: "12986e92b039d343ecbc4ec9bcaeaf78"
server_key_iv: "f5a95af36873536da35058586babb4d8"
server_key_version: "encryptionkey@0"
I20260923 14:36:17.366775 3689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "13d7e8ff8c9948d6b78480fd747cc6f1"
format_stamp: "Formatted at 2026-09-23 14:36:17 on dist-test-slave-zl47"
server_key: "12986e92b039d343ecbc4ec9bcaeaf78"
server_key_iv: "f5a95af36873536da35058586babb4d8"
server_key_version: "encryptionkey@0"
I20260923 14:36:17.369881 3689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:36:17.371917 3704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:17.372946 3689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:36:17.373097 3689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "13d7e8ff8c9948d6b78480fd747cc6f1"
format_stamp: "Formatted at 2026-09-23 14:36:17 on dist-test-slave-zl47"
server_key: "12986e92b039d343ecbc4ec9bcaeaf78"
server_key_iv: "f5a95af36873536da35058586babb4d8"
server_key_version: "encryptionkey@0"
I20260923 14:36:17.373200 3689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:17.384245 3689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:17.386775 3689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:17.386960 3689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:17.387465 3689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:17.388401 3689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:17.388489 3689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:17.388557 3689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:17.388612 3689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:17.397341 3689 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:38907
I20260923 14:36:17.397363 3817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:38907 every 8 connection(s)
I20260923 14:36:17.398227 3689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174177, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:17.407397 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 3689
I20260923 14:36:17.407521 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:17.407773 2920 external_mini_cluster.cc:1428] Setting key 38b244b89a13f969c69664e396848552
I20260923 14:36:17.410156 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.399773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:42833 (local address 127.2.218.62:43105)
0923 14:36:17.399899 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.399903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.400540 (+ 637us) server_negotiation.cc:407] Connection header received
0923 14:36:17.401756 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.401760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.401818 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.401921 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:17.402936 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.403417 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.404405 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.404625 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.404731 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.406924 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:17.406934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:17.406937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:17.406959 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:17.408419 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.408845 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.408849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.408851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.408927 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.409237 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.409242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.409243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.409402 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:17.409498 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.409898 (+ 400us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.409997 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:36:17.410990 3820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.400032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:43105 (local address 127.2.218.2:42833)
0923 14:36:17.400402 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:17.400436 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:17.401552 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:17.402086 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:17.402128 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:17.402426 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:17.402796 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.402807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.403537 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.403540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.404281 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.404289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.404760 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.404763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.404913 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.405315 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:17.405329 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:17.406731 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:17.408510 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:17.408515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:17.408526 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:17.408748 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:17.409013 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:17.409016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:17.409018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:17.409150 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:17.409496 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:17.409501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:17.409695 (+ 194us) client_negotiation.cc:769] Sending connection context
0923 14:36:17.409859 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 14:36:17.410127 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
I20260923 14:36:17.411996 3818 heartbeater.cc:347] Connected to a master server at 127.2.218.62:43105
I20260923 14:36:17.412201 3818 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:17.412585 3818 heartbeater.cc:511] Master 127.2.218.62:43105 requested a full tablet report, sending...
I20260923 14:36:17.413516 3490 ts_manager.cc:194] Registered new tserver with Master: 13d7e8ff8c9948d6b78480fd747cc6f1 (127.2.218.2:38907)
I20260923 14:36:17.414041 3490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:42833
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:36:17.430455 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:43105
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:17.523941 3824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:17.524166 3824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:17.524237 3824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:17.527604 3824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:17.527673 3824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:17.527753 3824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:17.532207 3824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:43105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:17.532984 3824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:17.533778 3824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:17.541556 3829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:17.541579 3830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:17.541697 3824 server_base.cc:1034] running on GCE node
W20260923 14:36:17.541824 3832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:17.542246 3824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:17.542732 3824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:17.543874 3824 hybrid_clock.cc:648] HybridClock initialized: now 1790174177543853 us; error 33 us; skew 500 ppm
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174177, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:17.546253 3824 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:36:17.546932 3824 webserver.cc:460] Webserver started at http://127.2.218.3:35773/ using document root <none> and password file <none>
I20260923 14:36:17.547370 3824 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:17.547446 3824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:17.547617 3824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:17.549208 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "829352a267e64375b7836e8580d0ea9a"
format_stamp: "Formatted at 2026-09-23 14:36:17 on dist-test-slave-zl47"
server_key: "b373f2fc1865aaa917351e91410624fe"
server_key_iv: "5a0db9904276d15a861184702767bc5e"
server_key_version: "encryptionkey@0"
I20260923 14:36:17.549674 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "829352a267e64375b7836e8580d0ea9a"
format_stamp: "Formatted at 2026-09-23 14:36:17 on dist-test-slave-zl47"
server_key: "b373f2fc1865aaa917351e91410624fe"
server_key_iv: "5a0db9904276d15a861184702767bc5e"
server_key_version: "encryptionkey@0"
I20260923 14:36:17.552562 3824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:17.554528 3839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:17.555567 3824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:17.555645 3824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "829352a267e64375b7836e8580d0ea9a"
format_stamp: "Formatted at 2026-09-23 14:36:17 on dist-test-slave-zl47"
server_key: "b373f2fc1865aaa917351e91410624fe"
server_key_iv: "5a0db9904276d15a861184702767bc5e"
server_key_version: "encryptionkey@0"
I20260923 14:36:17.555750 3824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:17.573288 3824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:17.576063 3824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:17.576218 3824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:17.576669 3824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:17.577361 3824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:17.577399 3824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:17.577448 3824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:17.577476 3824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:17.584769 3824 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:38173
I20260923 14:36:17.584820 3952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:38173 every 8 connection(s)
I20260923 14:36:17.585458 3824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174177, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:17.594709 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 3824
I20260923 14:36:17.594796 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:17.595013 2920 external_mini_cluster.cc:1428] Setting key 9959d8d6324f80833d1f34bb6b2c0ed4
I20260923 14:36:17.596422 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.586938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:33509 (local address 127.2.218.62:43105)
0923 14:36:17.587056 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.587065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.587667 (+ 602us) server_negotiation.cc:407] Connection header received
0923 14:36:17.588681 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.588683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.588730 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.588802 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:17.589661 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.590185 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.591085 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.591275 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.591347 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.593231 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:17.593235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:17.593236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:17.593257 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:17.594527 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.595077 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.595084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.595085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.595164 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.595480 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.595484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.595485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.595655 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:17.595755 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.596162 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.596266 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:36:17.597074 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.587162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:43105 (local address 127.2.218.3:33509)
0923 14:36:17.587510 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:17.587543 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:17.588499 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:17.588923 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:17.588928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:17.589187 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:17.589535 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.589544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.590279 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.590281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.590985 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.590992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.591379 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.591381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.591494 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.591881 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:17.591894 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:17.593072 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:17.594649 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:17.594654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:17.594665 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:17.594976 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:17.595254 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:17.595258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:17.595259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:17.595383 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:17.595753 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:17.595759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:17.595956 (+ 197us) client_negotiation.cc:769] Sending connection context
0923 14:36:17.596121 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 14:36:17.596368 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":106,"threads_started":1}
I20260923 14:36:17.597991 3953 heartbeater.cc:347] Connected to a master server at 127.2.218.62:43105
I20260923 14:36:17.598215 3953 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:17.598608 3953 heartbeater.cc:511] Master 127.2.218.62:43105 requested a full tablet report, sending...
I20260923 14:36:17.599450 3490 ts_manager.cc:194] Registered new tserver with Master: 829352a267e64375b7836e8580d0ea9a (127.2.218.3:38173)
I20260923 14:36:17.599901 3490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:33509
I20260923 14:36:17.608184 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:36:17.616737 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.611114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.2.218.62:43105)
0923 14:36:17.611278 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.611282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.611437 (+ 155us) server_negotiation.cc:407] Connection header received
0923 14:36:17.611511 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.611513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.611551 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.611611 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:17.612184 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.612599 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.613223 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.613389 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.613450 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.614466 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:17.614468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:17.614470 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:17.614487 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:17.615621 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.615913 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.615915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.615916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.615972 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.616147 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.616148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.616149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.616297 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:17.616380 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.616511 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.616588 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 14:36:17.620381 3490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:17.622879 3490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:17.635227 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.630049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:38173 (local address 127.0.0.1:36638)
0923 14:36:17.630550 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:17.630614 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:17.630738 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:17.631376 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:17.631380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:17.631399 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:17.631513 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.631518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.632663 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.632666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.633834 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.633844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.634764 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.634767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.634873 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.634906 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 14:36:17.635007 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 14:36:17.635089 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":21,"thread_start_us":77,"threads_started":1}
I20260923 14:36:17.635478 3966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.630139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.2.218.3:38173)
0923 14:36:17.630945 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.630958 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.630977 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:36:17.631036 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.631042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.631236 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.631382 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:17.631617 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.632539 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.633970 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.634647 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.634768 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.634887 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.635279 (+ 392us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.635352 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":112,"threads_started":1}
I20260923 14:36:17.635551 3968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.630478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:38907 (local address 127.0.0.1:33354)
0923 14:36:17.630925 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:17.630936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:17.631006 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:17.631819 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:17.631822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:17.631837 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:17.631910 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.631915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.633009 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.633012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.633864 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.633873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.634960 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.634963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.635056 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.635076 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:36:17.635155 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 14:36:17.635215 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":112,"threads_started":1}
I20260923 14:36:17.635812 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.630874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.2.218.2:38907)
0923 14:36:17.631257 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.631267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.631283 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:36:17.631347 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.631353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.631521 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.631670 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:17.632017 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.632888 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.633985 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.634758 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.634841 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.634958 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.635199 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.635283 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":119,"threads_started":1}
I20260923 14:36:17.636088 3969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.630736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:35737 (local address 127.0.0.1:40656)
0923 14:36:17.630971 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:17.630985 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:17.631149 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:17.632221 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:17.632223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:17.632235 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:17.632350 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.632355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.633580 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.633583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.634596 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.634606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.635713 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.635717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.635803 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.635830 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 14:36:17.635914 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 14:36:17.635984 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260923 14:36:17.636243 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.630380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.2.218.1:35737)
0923 14:36:17.630790 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.630802 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.631062 (+ 260us) server_negotiation.cc:407] Connection header received
0923 14:36:17.631829 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.631834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.631994 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.632092 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:17.632441 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.633434 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.634714 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.635438 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.635527 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.635655 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.636001 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.636087 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":139,"threads_started":1}
I20260923 14:36:17.637864 3887 tablet_service.cc:1467] Processing CreateTablet for tablet caac4eb992d14d5c9e7c47eab0b312a0 (DEFAULT_TABLE table=test-table [id=a29c07da9d9943bbb317a02e1f62b725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:17.637986 3752 tablet_service.cc:1467] Processing CreateTablet for tablet caac4eb992d14d5c9e7c47eab0b312a0 (DEFAULT_TABLE table=test-table [id=a29c07da9d9943bbb317a02e1f62b725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:17.638330 3617 tablet_service.cc:1467] Processing CreateTablet for tablet caac4eb992d14d5c9e7c47eab0b312a0 (DEFAULT_TABLE table=test-table [id=a29c07da9d9943bbb317a02e1f62b725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:17.638805 3887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caac4eb992d14d5c9e7c47eab0b312a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:17.638859 3752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caac4eb992d14d5c9e7c47eab0b312a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:17.639173 3617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caac4eb992d14d5c9e7c47eab0b312a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:17.644227 3971 tablet_bootstrap.cc:492] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a: Bootstrap starting.
I20260923 14:36:17.645635 3973 tablet_bootstrap.cc:492] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45: Bootstrap starting.
I20260923 14:36:17.646180 3971 tablet_bootstrap.cc:654] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:17.646349 3972 tablet_bootstrap.cc:492] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1: Bootstrap starting.
I20260923 14:36:17.647194 3971 log.cc:824] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:17.647779 3973 tablet_bootstrap.cc:654] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:17.648535 3972 tablet_bootstrap.cc:654] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:17.648584 3973 log.cc:824] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:17.648711 3971 tablet_bootstrap.cc:492] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a: No bootstrap required, opened a new log
I20260923 14:36:17.648880 3971 ts_tablet_manager.cc:1397] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:36:17.649199 3972 log.cc:824] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:17.650305 3973 tablet_bootstrap.cc:492] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45: No bootstrap required, opened a new log
I20260923 14:36:17.650454 3973 ts_tablet_manager.cc:1397] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:36:17.650800 3972 tablet_bootstrap.cc:492] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1: No bootstrap required, opened a new log
I20260923 14:36:17.650977 3972 ts_tablet_manager.cc:1397] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:36:17.651852 3971 raft_consensus.cc:348] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829352a267e64375b7836e8580d0ea9a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38173 } } peers { permanent_uuid: "9dcc45ed27694f17b07bf823587d5b45" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35737 } } peers { permanent_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 38907 } }
I20260923 14:36:17.652055 3971 raft_consensus.cc:374] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:17.652143 3971 raft_consensus.cc:729] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 829352a267e64375b7836e8580d0ea9a, State: Initialized, Role: FOLLOWER
I20260923 14:36:17.652539 3971 consensus_queue.cc:260] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829352a267e64375b7836e8580d0ea9a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38173 } } peers { permanent_uuid: "9dcc45ed27694f17b07bf823587d5b45" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35737 } } peers { permanent_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 38907 } }
I20260923 14:36:17.653040 3953 heartbeater.cc:503] Master 127.2.218.62:43105 was elected leader, sending a full tablet report...
I20260923 14:36:17.653337 3971 ts_tablet_manager.cc:1428] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260923 14:36:17.653606 3973 raft_consensus.cc:348] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829352a267e64375b7836e8580d0ea9a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38173 } } peers { permanent_uuid: "9dcc45ed27694f17b07bf823587d5b45" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35737 } } peers { permanent_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 38907 } }
I20260923 14:36:17.653827 3973 raft_consensus.cc:374] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:17.653877 3973 raft_consensus.cc:729] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcc45ed27694f17b07bf823587d5b45, State: Initialized, Role: FOLLOWER
I20260923 14:36:17.654028 3972 raft_consensus.cc:348] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829352a267e64375b7836e8580d0ea9a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38173 } } peers { permanent_uuid: "9dcc45ed27694f17b07bf823587d5b45" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35737 } } peers { permanent_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 38907 } }
I20260923 14:36:17.654250 3972 raft_consensus.cc:374] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:17.654253 3973 consensus_queue.cc:260] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829352a267e64375b7836e8580d0ea9a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38173 } } peers { permanent_uuid: "9dcc45ed27694f17b07bf823587d5b45" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35737 } } peers { permanent_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 38907 } }
I20260923 14:36:17.654330 3972 raft_consensus.cc:729] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d7e8ff8c9948d6b78480fd747cc6f1, State: Initialized, Role: FOLLOWER
I20260923 14:36:17.654676 3972 consensus_queue.cc:260] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829352a267e64375b7836e8580d0ea9a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38173 } } peers { permanent_uuid: "9dcc45ed27694f17b07bf823587d5b45" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35737 } } peers { permanent_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 38907 } }
I20260923 14:36:17.655028 3973 ts_tablet_manager.cc:1428] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 14:36:17.655200 3818 heartbeater.cc:503] Master 127.2.218.62:43105 was elected leader, sending a full tablet report...
I20260923 14:36:17.655421 3972 ts_tablet_manager.cc:1428] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 14:36:17.655989 3684 heartbeater.cc:503] Master 127.2.218.62:43105 was elected leader, sending a full tablet report...
W20260923 14:36:17.716442 3685 tablet.cc:2369] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:17.762249 3977 raft_consensus.cc:482] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:17.762436 3977 raft_consensus.cc:504] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829352a267e64375b7836e8580d0ea9a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38173 } } peers { permanent_uuid: "9dcc45ed27694f17b07bf823587d5b45" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35737 } } peers { permanent_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 38907 } }
I20260923 14:36:17.763475 3977 leader_election.cc:288] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dcc45ed27694f17b07bf823587d5b45 (127.2.218.1:35737), 13d7e8ff8c9948d6b78480fd747cc6f1 (127.2.218.2:38907)
I20260923 14:36:17.767040 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.763789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:42951 (local address 127.2.218.1:35737)
0923 14:36:17.763901 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.763905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.763918 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:36:17.763962 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.763965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.764009 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.764084 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:17.764407 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.764922 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.766091 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.766518 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.766596 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.766640 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.766861 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.766932 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:36:17.767043 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.763629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:35737 (local address 127.2.218.3:42951)
0923 14:36:17.763730 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:17.763743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:17.763820 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:17.764111 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:17.764117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:17.764141 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:17.764289 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.764297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.765019 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.765022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.765961 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.765971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.766622 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.766628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.766732 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.766753 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 14:36:17.766832 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 14:36:17.766905 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":53}
I20260923 14:36:17.767071 3980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.763712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:38907 (local address 127.2.218.3:43003)
0923 14:36:17.764040 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:17.764053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:17.764161 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:17.764406 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:17.764408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:17.764422 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:17.764521 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.764526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.765225 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.765228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.766003 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:17.766011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.766725 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.766728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:17.766794 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.766835 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 14:36:17.766909 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 14:36:17.766961 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":96,"threads_started":1}
I20260923 14:36:17.767122 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.763810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:43003 (local address 127.2.218.2:38907)
0923 14:36:17.763929 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.763934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.764141 (+ 207us) server_negotiation.cc:407] Connection header received
0923 14:36:17.764258 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.764262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.764310 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.764385 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:17.764618 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.765092 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.766186 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.766605 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.766707 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.766764 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.766940 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.767002 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:36:17.767791 3637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac4eb992d14d5c9e7c47eab0b312a0" candidate_uuid: "829352a267e64375b7836e8580d0ea9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc45ed27694f17b07bf823587d5b45" is_pre_election: true
I20260923 14:36:17.767812 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac4eb992d14d5c9e7c47eab0b312a0" candidate_uuid: "829352a267e64375b7836e8580d0ea9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" is_pre_election: true
I20260923 14:36:17.768066 3637 raft_consensus.cc:2454] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 829352a267e64375b7836e8580d0ea9a in term 0.
I20260923 14:36:17.768065 3772 raft_consensus.cc:2454] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 829352a267e64375b7836e8580d0ea9a in term 0.
I20260923 14:36:17.768474 3840 leader_election.cc:302] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d7e8ff8c9948d6b78480fd747cc6f1, 829352a267e64375b7836e8580d0ea9a; no voters:
I20260923 14:36:17.768724 3977 raft_consensus.cc:2781] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:17.768811 3977 raft_consensus.cc:482] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:17.768853 3977 raft_consensus.cc:3037] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:17.769788 3977 raft_consensus.cc:504] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829352a267e64375b7836e8580d0ea9a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38173 } } peers { permanent_uuid: "9dcc45ed27694f17b07bf823587d5b45" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35737 } } peers { permanent_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 38907 } }
I20260923 14:36:17.770124 3977 leader_election.cc:288] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [CANDIDATE]: Term 1 election: Requested vote from peers 9dcc45ed27694f17b07bf823587d5b45 (127.2.218.1:35737), 13d7e8ff8c9948d6b78480fd747cc6f1 (127.2.218.2:38907)
I20260923 14:36:17.770323 3637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac4eb992d14d5c9e7c47eab0b312a0" candidate_uuid: "829352a267e64375b7836e8580d0ea9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc45ed27694f17b07bf823587d5b45"
I20260923 14:36:17.770437 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac4eb992d14d5c9e7c47eab0b312a0" candidate_uuid: "829352a267e64375b7836e8580d0ea9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1"
I20260923 14:36:17.770459 3637 raft_consensus.cc:3037] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:17.770555 3772 raft_consensus.cc:3037] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:17.771606 3637 raft_consensus.cc:2454] T caac4eb992d14d5c9e7c47eab0b312a0 P 9dcc45ed27694f17b07bf823587d5b45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829352a267e64375b7836e8580d0ea9a in term 1.
I20260923 14:36:17.771602 3772 raft_consensus.cc:2454] T caac4eb992d14d5c9e7c47eab0b312a0 P 13d7e8ff8c9948d6b78480fd747cc6f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 829352a267e64375b7836e8580d0ea9a in term 1.
I20260923 14:36:17.771869 3843 leader_election.cc:302] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 829352a267e64375b7836e8580d0ea9a, 9dcc45ed27694f17b07bf823587d5b45; no voters:
I20260923 14:36:17.772058 3977 raft_consensus.cc:2781] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:17.772241 3977 raft_consensus.cc:686] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [term 1 LEADER]: Becoming Leader. State: Replica: 829352a267e64375b7836e8580d0ea9a, State: Running, Role: LEADER
I20260923 14:36:17.772522 3977 consensus_queue.cc:237] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829352a267e64375b7836e8580d0ea9a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38173 } } peers { permanent_uuid: "9dcc45ed27694f17b07bf823587d5b45" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35737 } } peers { permanent_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 38907 } }
I20260923 14:36:17.775422 3489 catalog_manager.cc:5495] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 829352a267e64375b7836e8580d0ea9a (127.2.218.3). New cstate: current_term: 1 leader_uuid: "829352a267e64375b7836e8580d0ea9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "829352a267e64375b7836e8580d0ea9a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dcc45ed27694f17b07bf823587d5b45" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d7e8ff8c9948d6b78480fd747cc6f1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 38907 } health_report { overall_health: UNKNOWN } } }
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:36:17.790887 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.782737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.2.218.1:35737)
0923 14:36:17.782881 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.782887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.783155 (+ 268us) server_negotiation.cc:407] Connection header received
0923 14:36:17.783280 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.783283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.783327 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.783404 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:17.784090 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.784489 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.785353 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.785569 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.785680 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.788107 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:17.788115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:17.788124 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:17.788142 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:17.789363 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.789721 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.789725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.789726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.789817 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.790042 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.790044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.790045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.790342 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:17.790475 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.790641 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.790735 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:36:17.813448 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.805885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.2.218.1:35737)
0923 14:36:17.806013 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.806018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.806178 (+ 160us) server_negotiation.cc:407] Connection header received
0923 14:36:17.806300 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.806303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.806345 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.806425 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:17.806996 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.807402 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.808112 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.808301 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.808401 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.810511 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:17.810515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:17.810517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:17.810540 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:17.811929 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.812316 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.812321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.812322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.812397 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.812669 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.812672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.812674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.812833 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:17.812960 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.813206 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.813310 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:36:17.833806 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:17.826984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.2.218.1:35737)
0923 14:36:17.827120 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:17.827125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:17.827257 (+ 132us) server_negotiation.cc:407] Connection header received
0923 14:36:17.827392 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:17.827395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:17.827434 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:17.827512 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:17.828107 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.828498 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.829225 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:17.829408 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:17.829508 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:17.831612 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:17.831615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:17.831616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:17.831633 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:17.832622 (+ 989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.833013 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.833016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.833017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.833085 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:17.833294 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:17.833295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:17.833296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:17.833413 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:17.833493 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:17.833648 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 14:36:17.833718 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 14:36:17.834942 3596 tablet_service.cc:2908] Rejecting scan request for tablet caac4eb992d14d5c9e7c47eab0b312a0: Uninitialized: clean time has not yet been initialized
W20260923 14:36:17.836277 3954 tablet.cc:2369] T caac4eb992d14d5c9e7c47eab0b312a0 P 829352a267e64375b7836e8580d0ea9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:17.836839 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 3553
I20260923 14:36:17.842265 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 3689
I20260923 14:36:17.846903 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 3824
I20260923 14:36:17.851440 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 3458
2026-09-23T14:36:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2039 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3995](info): set up 2 sockets
Sep 23 14:36:17 dist-test-slave-zl47 krb5kdc[3995](info): commencing operation
krb5kdc: starting...
W20260923 14:36:18.762271 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.890s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:18 dist-test-slave-zl47 krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:18.869277 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33117
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:18.968600 4008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:18.968796 4008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:18.968829 4008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:18.972581 4008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:18.972628 4008 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 14:36:18.972645 4008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:18.972690 4008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:18.972721 4008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:18.972747 4008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:18.977648 4008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:33117
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:18.978477 4008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:18.979282 4008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:18.986757 4013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:18.986905 4008 server_base.cc:1034] running on GCE node
W20260923 14:36:18.986770 4016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:18.987074 4014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:18.987663 4008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:18.988768 4008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:18.989966 4008 hybrid_clock.cc:648] HybridClock initialized: now 1790174178989953 us; error 33 us; skew 500 ppm
Sep 23 14:36:18 dist-test-slave-zl47 krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:18.996053 4008 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:18.996915 4008 webserver.cc:460] Webserver started at http://127.2.218.62:33571/ using document root <none> and password file <none>
I20260923 14:36:18.997450 4008 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:18.997538 4008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:18.997758 4008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:18.999673 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "b30290f65ac04e11a88bb60a254f6b1b"
format_stamp: "Formatted at 2026-09-23 14:36:18 on dist-test-slave-zl47"
server_key: "bc93322ee5fd89e6ea58dba73dfc6c87"
server_key_iv: "f8e6a705b7ca048c984da2bb6ed1ba80"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.000252 4008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "b30290f65ac04e11a88bb60a254f6b1b"
format_stamp: "Formatted at 2026-09-23 14:36:18 on dist-test-slave-zl47"
server_key: "bc93322ee5fd89e6ea58dba73dfc6c87"
server_key_iv: "f8e6a705b7ca048c984da2bb6ed1ba80"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.003876 4008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:19.006422 4023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:19.007844 4008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:36:19.007992 4008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "b30290f65ac04e11a88bb60a254f6b1b"
format_stamp: "Formatted at 2026-09-23 14:36:18 on dist-test-slave-zl47"
server_key: "bc93322ee5fd89e6ea58dba73dfc6c87"
server_key_iv: "f8e6a705b7ca048c984da2bb6ed1ba80"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.008098 4008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:19.022259 4008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:19.024834 4008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:19.025025 4008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:19.033345 4008 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:33117
I20260923 14:36:19.033572 4085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:33117 every 8 connection(s)
I20260923 14:36:19.034368 4008 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:19.036746 4086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:19.041675 4086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b: Bootstrap starting.
I20260923 14:36:19.043851 4086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:19.044170 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 4008
I20260923 14:36:19.044286 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:19.044533 2920 external_mini_cluster.cc:1428] Setting key 96b91804cfd7a3ccc072f18d17d646ad
I20260923 14:36:19.044674 4086 log.cc:824] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:19.046980 4086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b: No bootstrap required, opened a new log
I20260923 14:36:19.050909 4086 raft_consensus.cc:348] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30290f65ac04e11a88bb60a254f6b1b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33117 } }
I20260923 14:36:19.051107 4086 raft_consensus.cc:374] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:19.051211 4086 raft_consensus.cc:729] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b30290f65ac04e11a88bb60a254f6b1b, State: Initialized, Role: FOLLOWER
I20260923 14:36:19.051751 4086 consensus_queue.cc:260] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30290f65ac04e11a88bb60a254f6b1b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33117 } }
I20260923 14:36:19.051903 4086 raft_consensus.cc:388] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:19.051990 4086 raft_consensus.cc:482] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:19.052086 4086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [term 0 FOLLOWER]: Advancing to term 1
Sep 23 14:36:19 dist-test-slave-zl47 krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:19.057418 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.045617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.2.218.62:33117)
0923 14:36:19.045977 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.045994 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.046038 (+ 44us) server_negotiation.cc:407] Connection header received
0923 14:36:19.046805 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.046826 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.047193 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.047516 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:19.048274 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.049172 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.050039 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.050589 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.050769 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.053518 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:19.053523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:19.053534 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:19.053568 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:19.055057 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.055437 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.055441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.055445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.055546 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.055842 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.055844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.055845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.056181 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:19.056308 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.056506 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.056719 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":124,"threads_started":1}
I20260923 14:36:19.108013 4086 raft_consensus.cc:504] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30290f65ac04e11a88bb60a254f6b1b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33117 } }
I20260923 14:36:19.108459 4086 leader_election.cc:302] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b30290f65ac04e11a88bb60a254f6b1b; no voters:
I20260923 14:36:19.108814 4086 leader_election.cc:288] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:19.108954 4091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:19.109233 4091 raft_consensus.cc:686] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [term 1 LEADER]: Becoming Leader. State: Replica: b30290f65ac04e11a88bb60a254f6b1b, State: Running, Role: LEADER
I20260923 14:36:19.109597 4091 consensus_queue.cc:237] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30290f65ac04e11a88bb60a254f6b1b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33117 } }
I20260923 14:36:19.110242 4086 sys_catalog.cc:564] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:19.111414 4092 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b30290f65ac04e11a88bb60a254f6b1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30290f65ac04e11a88bb60a254f6b1b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33117 } } }
I20260923 14:36:19.111567 4092 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:19.111413 4093 sys_catalog.cc:455] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b30290f65ac04e11a88bb60a254f6b1b. Latest consensus state: current_term: 1 leader_uuid: "b30290f65ac04e11a88bb60a254f6b1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b30290f65ac04e11a88bb60a254f6b1b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33117 } } }
I20260923 14:36:19.111634 4093 sys_catalog.cc:458] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:19.112624 4100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:19.115208 4100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:19.120134 4100 catalog_manager.cc:1347] Generated new cluster ID: 2aa7c3e738624f07959c888f15da28cd
I20260923 14:36:19.120193 4100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:19.131207 4100 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:19.132229 4100 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:19.141103 4100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b: Generated new TSK 0
I20260923 14:36:19.141723 4100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:36:19.180450 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33117
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:19.277290 4113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:19.277514 4113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:19.277587 4113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:19.281149 4113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:19.281224 4113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:19.281288 4113 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:36:19.281361 4113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:19.285986 4113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.218.62:33117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:19.286782 4113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:19.287611 4113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:19.295862 4121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:19.295949 4118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:19.295907 4119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:19.296034 4113 server_base.cc:1034] running on GCE node
I20260923 14:36:19.296492 4113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:19.297063 4113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:19.298250 4113 hybrid_clock.cc:648] HybridClock initialized: now 1790174179298219 us; error 36 us; skew 500 ppm
Sep 23 14:36:19 dist-test-slave-zl47 krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174179, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:19.300724 4113 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:36:19.301548 4113 webserver.cc:460] Webserver started at http://127.2.218.1:34581/ using document root <none> and password file <none>
I20260923 14:36:19.302057 4113 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:19.302171 4113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:19.302412 4113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:19.304188 4113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "19fe0fe6f9af40a58e2d9418d5f75244"
format_stamp: "Formatted at 2026-09-23 14:36:19 on dist-test-slave-zl47"
server_key: "d381e900c1069902b7d4a48ca7de04eb"
server_key_iv: "1dcd9fc9975aaa58f11f48ca9a7017ae"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.304688 4113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "19fe0fe6f9af40a58e2d9418d5f75244"
format_stamp: "Formatted at 2026-09-23 14:36:19 on dist-test-slave-zl47"
server_key: "d381e900c1069902b7d4a48ca7de04eb"
server_key_iv: "1dcd9fc9975aaa58f11f48ca9a7017ae"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.307897 4113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:36:19.310071 4128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:19.311440 4113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:19.311566 4113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "19fe0fe6f9af40a58e2d9418d5f75244"
format_stamp: "Formatted at 2026-09-23 14:36:19 on dist-test-slave-zl47"
server_key: "d381e900c1069902b7d4a48ca7de04eb"
server_key_iv: "1dcd9fc9975aaa58f11f48ca9a7017ae"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.311688 4113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:19.339751 4113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:19.342340 4113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:19.342520 4113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:19.343739 4113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:19.343811 4113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:19.343878 4113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:19.343921 4113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:36:19 dist-test-slave-zl47 krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174179, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:19.354991 4113 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:44519
I20260923 14:36:19.355837 4113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:19.356406 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.344428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:53777 (local address 127.2.218.62:33117)
0923 14:36:19.344553 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.344557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.345312 (+ 755us) server_negotiation.cc:407] Connection header received
0923 14:36:19.346384 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.346387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.346433 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.346510 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:19.347690 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.348431 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.349583 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.349792 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.349892 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.352508 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:19.352512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:19.352514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:19.352543 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:19.354431 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.354955 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.354959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.354961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.355036 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.355342 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.355347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.355348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.355509 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:19.355619 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.356148 (+ 529us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.356274 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:36:19.356805 4243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:44519 every 8 connection(s)
I20260923 14:36:19.357160 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.344736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33117 (local address 127.2.218.1:53777)
0923 14:36:19.345158 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.345206 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.346179 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.346749 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.346756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.347024 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:19.347561 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.347577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.348558 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.348562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.349471 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.349480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.349931 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.349935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.350075 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.350598 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:19.350617 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:19.352353 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:19.354551 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.354557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.354577 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.354858 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.355128 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.355133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.355134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.355257 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.355644 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:19.355650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:19.355909 (+ 259us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.356117 (+ 208us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.356407 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":84,"threads_started":1}
I20260923 14:36:19.358197 4244 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33117
I20260923 14:36:19.358341 4244 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:19.358737 4244 heartbeater.cc:511] Master 127.2.218.62:33117 requested a full tablet report, sending...
I20260923 14:36:19.360067 4039 ts_manager.cc:194] Registered new tserver with Master: 19fe0fe6f9af40a58e2d9418d5f75244 (127.2.218.1:44519)
I20260923 14:36:19.361095 4039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:53777
I20260923 14:36:19.365855 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 4113
I20260923 14:36:19.365868 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.359977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:56045 (local address 127.2.218.62:33117)
0923 14:36:19.360083 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.360086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.360165 (+ 79us) server_negotiation.cc:407] Connection header received
0923 14:36:19.360313 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.360315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.360373 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.360443 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:19.360809 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.361288 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.362088 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.362313 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.362408 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.363375 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:19.363378 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:19.363380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:19.363402 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:19.364519 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.364867 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.364871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.364872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.364937 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.365167 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.365172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.365173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.365308 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:19.365393 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.365650 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.365740 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 14:36:19.365898 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.359893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33117 (local address 127.2.218.1:56045)
0923 14:36:19.360100 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.360112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.360229 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.360457 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.360459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.360623 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:19.360699 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.360703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.361401 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.361404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.361975 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.361984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.362412 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.362416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.362498 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.362923 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:19.362933 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:19.363248 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:19.364617 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.364620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.364622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.364782 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.365011 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.365015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.365017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.365092 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.365387 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:19.365390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:19.365483 (+ 93us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.365599 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.365736 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260923 14:36:19.365988 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:19.366299 2920 external_mini_cluster.cc:1428] Setting key f9abc32aeb2cb3289dfe8ea68df42ec1
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:36:19.390450 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33117
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:19.488921 4253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:19.489123 4253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:19.489161 4253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:19.492846 4253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:19.492909 4253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:19.492949 4253 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:36:19.492976 4253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:19.497931 4253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.218.62:33117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:19.498987 4253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:19.499852 4253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:19.508248 4259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:19.508292 4253 server_base.cc:1034] running on GCE node
W20260923 14:36:19.508257 4261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:19.508245 4258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:19.508814 4253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:19.509325 4253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:19.510494 4253 hybrid_clock.cc:648] HybridClock initialized: now 1790174179510476 us; error 30 us; skew 500 ppm
Sep 23 14:36:19 dist-test-slave-zl47 krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174179, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:19.513118 4253 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:36:19.513762 4253 webserver.cc:460] Webserver started at http://127.2.218.2:34437/ using document root <none> and password file <none>
I20260923 14:36:19.514240 4253 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:19.514297 4253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:19.514432 4253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:19.516249 4253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "c091da166fad47dbac8b5f99e17a444f"
format_stamp: "Formatted at 2026-09-23 14:36:19 on dist-test-slave-zl47"
server_key: "7ec9be75a6bb8cc9e0d2611904bdefc1"
server_key_iv: "4165c25d99445bc2e99bd3904dfbd5a9"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.516733 4253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "c091da166fad47dbac8b5f99e17a444f"
format_stamp: "Formatted at 2026-09-23 14:36:19 on dist-test-slave-zl47"
server_key: "7ec9be75a6bb8cc9e0d2611904bdefc1"
server_key_iv: "4165c25d99445bc2e99bd3904dfbd5a9"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.519713 4253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:19.521690 4268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:19.522761 4253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:19.522863 4253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "c091da166fad47dbac8b5f99e17a444f"
format_stamp: "Formatted at 2026-09-23 14:36:19 on dist-test-slave-zl47"
server_key: "7ec9be75a6bb8cc9e0d2611904bdefc1"
server_key_iv: "4165c25d99445bc2e99bd3904dfbd5a9"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.522949 4253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:19.530717 4253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:19.533195 4253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:19.533370 4253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:19.534665 4253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:19.534713 4253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:19.534744 4253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:19.534790 4253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:36:19 dist-test-slave-zl47 krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174179, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:19.544255 4253 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:45405
I20260923 14:36:19.544569 4384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:45405 every 8 connection(s)
I20260923 14:36:19.545084 4253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:36:19.551357 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.535241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:40587 (local address 127.2.218.62:33117)
0923 14:36:19.535389 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.535395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.536518 (+ 1123us) server_negotiation.cc:407] Connection header received
0923 14:36:19.537651 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.537656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.537717 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.537812 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:19.539100 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.539976 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.541318 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.541512 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.541614 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.543976 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:19.543979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:19.543982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:19.544006 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:19.545581 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.546220 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.546224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.546227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.546322 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.550279 (+ 3957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.550284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.550287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.550509 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:19.550582 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.551161 (+ 579us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.551245 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:36:19.552326 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.535928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33117 (local address 127.2.218.2:40587)
0923 14:36:19.536361 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.536406 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.537417 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.537945 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.537951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.538401 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:19.538948 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.538965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.540122 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.540128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.541199 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.541208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.541639 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.541642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.541793 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.542295 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:19.542308 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:19.543817 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:19.545774 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.545778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.545790 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.546057 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.549887 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.549891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.549894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.550078 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.550611 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:19.550620 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:19.550919 (+ 299us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.551122 (+ 203us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.551452 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":148,"threads_started":1}
I20260923 14:36:19.553501 4385 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33117
I20260923 14:36:19.553622 4385 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:19.553979 4385 heartbeater.cc:511] Master 127.2.218.62:33117 requested a full tablet report, sending...
I20260923 14:36:19.554939 4040 ts_manager.cc:194] Registered new tserver with Master: c091da166fad47dbac8b5f99e17a444f (127.2.218.2:45405)
I20260923 14:36:19.555318 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 4253
I20260923 14:36:19.555428 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:19.555672 4040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:40587
I20260923 14:36:19.555742 2920 external_mini_cluster.cc:1428] Setting key 54e3945f8c91a6e3caf84b332e97c5eb
I20260923 14:36:19.564109 4391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.556158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33117 (local address 127.2.218.2:53481)
0923 14:36:19.556632 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.556649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.556779 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.557175 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.557179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.557409 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:19.557537 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.557545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.558538 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.558542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.559370 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.559381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.560007 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.560010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.560075 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.560503 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:19.560513 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:19.560766 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:19.562492 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.562497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.562502 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.562734 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.563018 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.563023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.563027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.563118 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.563647 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:19.563650 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:19.563754 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.563870 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.563978 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":260,"threads_started":1}
I20260923 14:36:19.564304 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.555970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:53481 (local address 127.2.218.62:33117)
0923 14:36:19.556066 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.556070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.556767 (+ 697us) server_negotiation.cc:407] Connection header received
0923 14:36:19.556909 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.556915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.556963 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.557017 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:19.557674 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.558368 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.559519 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.559880 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.559988 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.560904 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:19.560910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:19.560914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:19.560941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:19.562388 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.562832 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.562838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.562840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.562920 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.563226 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.563231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.563235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.563498 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:19.563632 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.563950 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.564086 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:36:19.585012 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33117
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:19.688184 4394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:19.688469 4394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:19.688597 4394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:19.692518 4394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:19.692596 4394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:19.692701 4394 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:36:19.692817 4394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:19.697751 4394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.218.62:33117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:19.698714 4394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:19.699611 4394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:19.710608 4402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:19.710690 4394 server_base.cc:1034] running on GCE node
W20260923 14:36:19.710608 4400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:19.710913 4399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:19.711380 4394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:19.711998 4394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:19.713179 4394 hybrid_clock.cc:648] HybridClock initialized: now 1790174179713162 us; error 32 us; skew 500 ppm
Sep 23 14:36:19 dist-test-slave-zl47 krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174179, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:19.715911 4394 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:36:19.716622 4394 webserver.cc:460] Webserver started at http://127.2.218.3:44791/ using document root <none> and password file <none>
I20260923 14:36:19.717093 4394 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:19.717175 4394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:19.717376 4394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:19.719295 4394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "9a02238e43b544c0874193ee2406ae97"
format_stamp: "Formatted at 2026-09-23 14:36:19 on dist-test-slave-zl47"
server_key: "f260ae1535fc0e8d18657a12ae7d1194"
server_key_iv: "641e218fb26cfb23a4593a97c31f7187"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.719837 4394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "9a02238e43b544c0874193ee2406ae97"
format_stamp: "Formatted at 2026-09-23 14:36:19 on dist-test-slave-zl47"
server_key: "f260ae1535fc0e8d18657a12ae7d1194"
server_key_iv: "641e218fb26cfb23a4593a97c31f7187"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.723279 4394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:36:19.725584 4409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:19.726868 4394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:19.727065 4394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "9a02238e43b544c0874193ee2406ae97"
format_stamp: "Formatted at 2026-09-23 14:36:19 on dist-test-slave-zl47"
server_key: "f260ae1535fc0e8d18657a12ae7d1194"
server_key_iv: "641e218fb26cfb23a4593a97c31f7187"
server_key_version: "encryptionkey@0"
I20260923 14:36:19.727255 4394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:19.748270 4394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:19.750967 4394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:19.751197 4394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:19.752697 4394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:19.752794 4394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:19.752893 4394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:19.752938 4394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:36:19 dist-test-slave-zl47 krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174179, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:19.763916 4394 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:35669
I20260923 14:36:19.763962 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:35669 every 8 connection(s)
I20260923 14:36:19.764668 4394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:36:19.766014 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.753590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:53845 (local address 127.2.218.62:33117)
0923 14:36:19.753696 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.753700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.754690 (+ 990us) server_negotiation.cc:407] Connection header received
0923 14:36:19.755504 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.755507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.755569 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.755639 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:19.756906 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.757682 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.758957 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.759191 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.759293 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.762624 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:19.762627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:19.762629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:19.762654 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:19.764160 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.764643 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.764646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.764648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.764725 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.765026 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.765029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.765030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.765250 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:19.765351 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.765824 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.765912 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:36:19.766635 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.754061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33117 (local address 127.2.218.3:53845)
0923 14:36:19.754543 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.754596 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.755319 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.755784 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.755791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.756221 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:19.756768 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.756782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.757823 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.757827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.758829 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.758839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.759330 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.759334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.759534 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.760133 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:19.760158 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:19.762464 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:19.764265 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.764270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.764282 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.764530 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.764801 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.764803 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.764806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.764933 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.765398 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:19.765405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:19.765614 (+ 209us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.765740 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.765928 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":124,"threads_started":1}
I20260923 14:36:19.767697 4526 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33117
I20260923 14:36:19.767844 4526 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:19.768303 4526 heartbeater.cc:511] Master 127.2.218.62:33117 requested a full tablet report, sending...
I20260923 14:36:19.769239 4039 ts_manager.cc:194] Registered new tserver with Master: 9a02238e43b544c0874193ee2406ae97 (127.2.218.3:35669)
I20260923 14:36:19.769873 4039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:53845
I20260923 14:36:19.770124 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 4394
I20260923 14:36:19.770228 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:19.770526 2920 external_mini_cluster.cc:1428] Setting key d84a843f1fd624a7324f503884573bbe
I20260923 14:36:19.772365 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:36:19.776597 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.769688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:45473 (local address 127.2.218.62:33117)
0923 14:36:19.769768 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.769772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.769925 (+ 153us) server_negotiation.cc:407] Connection header received
0923 14:36:19.770052 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.770057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.770116 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.770182 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:19.770620 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.771175 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.771948 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.772219 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.772310 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.773410 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:19.773413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:19.773415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:19.773434 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:19.774883 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.775335 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.775337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.775338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.775409 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.775746 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.775752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.775754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.775926 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:19.776041 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.776329 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.776438 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 14:36:19.776607 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.769602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33117 (local address 127.2.218.3:45473)
0923 14:36:19.769824 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.769836 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.769941 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.770267 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.770270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.770436 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:19.770514 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.770519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.771305 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.771308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.771855 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.771862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.772349 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.772353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.772453 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.772820 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:19.772837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:19.773278 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:19.774989 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.774994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.774996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.775236 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.775544 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:19.775548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:19.775550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:19.775644 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:19.776030 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:19.776034 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:19.776150 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.776271 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.776405 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
Sep 23 14:36:19 dist-test-slave-zl47 krb5kdc[3995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 14:36:19 dist-test-slave-zl47 krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:19.796124 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.789423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.2.218.62:33117)
0923 14:36:19.789544 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.789547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.789556 (+ 9us) server_negotiation.cc:407] Connection header received
0923 14:36:19.789582 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.789583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.789612 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.789695 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:19.790323 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.790748 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.791480 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.791675 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.791768 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.793791 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:19.793794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:19.793795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:19.793813 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:19.794914 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.795280 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.795282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.795283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.795341 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:19.795553 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:19.795555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:19.795555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:19.795695 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:19.795778 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.795969 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.796039 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:36:19.798538 4039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:19.801106 4039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:19.814005 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.808931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.2.218.1:44519)
0923 14:36:19.809885 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.809898 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.809918 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:36:19.809985 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.809988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.810183 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.810628 (+ 445us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:19.810736 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.811505 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.812950 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.813427 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.813541 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.813686 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.813805 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.813889 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":102,"threads_started":1}
I20260923 14:36:19.813997 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.808411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:44519 (local address 127.0.0.1:59074)
0923 14:36:19.809051 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.809102 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.809218 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.810390 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.810393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.810407 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:19.810494 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.810501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.811646 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.811652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.812839 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.812848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.813541 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.813544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.813647 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.813685 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.813781 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.813878 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":114,"spinlock_wait_cycles":5888,"thread_start_us":81,"threads_started":1}
I20260923 14:36:19.814229 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.808240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:35669 (local address 127.0.0.1:55396)
0923 14:36:19.809051 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.809102 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.809218 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.809926 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.809931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.809954 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:19.810055 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.810082 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.811530 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.811535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.812769 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.812778 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.813852 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.813855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.813938 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.813960 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.814020 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.814091 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":77,"spinlock_wait_cycles":5120,"thread_start_us":65,"threads_started":1}
I20260923 14:36:19.814572 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.808364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.2.218.3:35669)
0923 14:36:19.809479 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.809489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.809510 (+ 21us) server_negotiation.cc:407] Connection header received
0923 14:36:19.809572 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.809577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.809751 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.810245 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:19.810368 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.811425 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.812876 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.813582 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.813672 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.813797 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.814376 (+ 579us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.814455 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":116,"threads_started":1}
I20260923 14:36:19.815896 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.808824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:45405 (local address 127.0.0.1:34892)
0923 14:36:19.809733 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.809747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.809836 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.810244 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.810248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.810265 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:19.810391 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.810397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.811734 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.811737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.813758 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.813767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.815525 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.815528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.815624 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.815646 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.815718 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.815777 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":4,"thread_start_us":99,"threads_started":1}
I20260923 14:36:19.816627 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.809273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.2.218.2:45405)
0923 14:36:19.809574 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.809584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.809835 (+ 251us) server_negotiation.cc:407] Connection header received
0923 14:36:19.809930 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.809934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.810114 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.810244 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:19.810528 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.811547 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.813876 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.814616 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.815335 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.815460 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.816059 (+ 599us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.816149 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
I20260923 14:36:19.816978 4459 tablet_service.cc:1467] Processing CreateTablet for tablet efbca0a9fbd54514a703407b0fdf5df4 (DEFAULT_TABLE table=test-table [id=9a1b1afc331f4428afd1b97efeb67612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:19.817010 4178 tablet_service.cc:1467] Processing CreateTablet for tablet efbca0a9fbd54514a703407b0fdf5df4 (DEFAULT_TABLE table=test-table [id=9a1b1afc331f4428afd1b97efeb67612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:19.817764 4178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbca0a9fbd54514a703407b0fdf5df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:19.817775 4459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbca0a9fbd54514a703407b0fdf5df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:19.823051 4549 tablet_bootstrap.cc:492] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: Bootstrap starting.
I20260923 14:36:19.823807 4550 tablet_bootstrap.cc:492] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: Bootstrap starting.
I20260923 14:36:19.824463 4318 tablet_service.cc:1467] Processing CreateTablet for tablet efbca0a9fbd54514a703407b0fdf5df4 (DEFAULT_TABLE table=test-table [id=9a1b1afc331f4428afd1b97efeb67612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:19.824856 4549 tablet_bootstrap.cc:654] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:19.825300 4318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbca0a9fbd54514a703407b0fdf5df4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:19.825423 4549 log.cc:824] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:19.825649 4550 tablet_bootstrap.cc:654] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:19.826398 4550 log.cc:824] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:19.827291 4549 tablet_bootstrap.cc:492] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: No bootstrap required, opened a new log
I20260923 14:36:19.827479 4549 ts_tablet_manager.cc:1397] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:36:19.828824 4550 tablet_bootstrap.cc:492] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: No bootstrap required, opened a new log
I20260923 14:36:19.829038 4550 ts_tablet_manager.cc:1397] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:36:19.829998 4549 raft_consensus.cc:348] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } }
I20260923 14:36:19.830269 4549 raft_consensus.cc:374] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:19.830318 4549 raft_consensus.cc:729] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fe0fe6f9af40a58e2d9418d5f75244, State: Initialized, Role: FOLLOWER
I20260923 14:36:19.830680 4549 consensus_queue.cc:260] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } }
I20260923 14:36:19.831167 4553 tablet_bootstrap.cc:492] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: Bootstrap starting.
I20260923 14:36:19.831630 4549 ts_tablet_manager.cc:1428] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 14:36:19.831976 4244 heartbeater.cc:503] Master 127.2.218.62:33117 was elected leader, sending a full tablet report...
I20260923 14:36:19.833156 4553 tablet_bootstrap.cc:654] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:19.833393 4550 raft_consensus.cc:348] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } }
I20260923 14:36:19.833571 4550 raft_consensus.cc:374] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:19.833622 4550 raft_consensus.cc:729] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a02238e43b544c0874193ee2406ae97, State: Initialized, Role: FOLLOWER
I20260923 14:36:19.833844 4553 log.cc:824] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:19.834038 4550 consensus_queue.cc:260] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } }
I20260923 14:36:19.834590 4526 heartbeater.cc:503] Master 127.2.218.62:33117 was elected leader, sending a full tablet report...
I20260923 14:36:19.834896 4550 ts_tablet_manager.cc:1428] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260923 14:36:19.835619 4553 tablet_bootstrap.cc:492] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: No bootstrap required, opened a new log
I20260923 14:36:19.835781 4553 ts_tablet_manager.cc:1397] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260923 14:36:19.838774 4553 raft_consensus.cc:348] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } }
I20260923 14:36:19.838932 4553 raft_consensus.cc:374] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:19.838960 4553 raft_consensus.cc:729] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c091da166fad47dbac8b5f99e17a444f, State: Initialized, Role: FOLLOWER
I20260923 14:36:19.839216 4553 consensus_queue.cc:260] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } }
I20260923 14:36:19.839987 4385 heartbeater.cc:503] Master 127.2.218.62:33117 was elected leader, sending a full tablet report...
I20260923 14:36:19.840219 4553 ts_tablet_manager.cc:1428] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
W20260923 14:36:19.857121 4245 tablet.cc:2369] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:19.881104 4557 raft_consensus.cc:482] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:19.881282 4557 raft_consensus.cc:504] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } }
I20260923 14:36:19.882341 4557 leader_election.cc:288] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a02238e43b544c0874193ee2406ae97 (127.2.218.3:35669), 19fe0fe6f9af40a58e2d9418d5f75244 (127.2.218.1:44519)
I20260923 14:36:19.886770 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.882735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:55517 (local address 127.2.218.1:44519)
0923 14:36:19.882830 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.882834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.882972 (+ 138us) server_negotiation.cc:407] Connection header received
0923 14:36:19.883052 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.883054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.883091 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.883142 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:19.883379 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.884026 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.885448 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.886121 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.886224 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.886282 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.886592 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.886653 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:36:19.886770 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.882604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:49701 (local address 127.2.218.3:35669)
0923 14:36:19.882715 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.882719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.882736 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:36:19.882785 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.882788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.882832 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.882909 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:19.883221 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.883941 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.885450 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.886121 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.886223 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.886282 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.886583 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.886652 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:36:19.886790 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.882501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:35669 (local address 127.2.218.2:49701)
0923 14:36:19.882603 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.882622 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.882700 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.882945 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.882947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.882964 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:19.883089 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.883094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.884034 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.884036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.885337 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.885347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.886232 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.886235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.886426 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.886479 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.886553 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.886651 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":170}
I20260923 14:36:19.886802 4558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.882544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:44519 (local address 127.2.218.2:55517)
0923 14:36:19.882868 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:19.882878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:19.882978 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:19.883178 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:19.883181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:19.883199 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:19.883290 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.883293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.884122 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.884125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.885337 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:19.885347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.886232 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.886235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:19.886402 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.886475 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 14:36:19.886553 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:36:19.886652 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":125,"spinlock_wait_cycles":2944,"thread_start_us":99,"threads_started":1}
I20260923 14:36:19.887394 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbca0a9fbd54514a703407b0fdf5df4" candidate_uuid: "c091da166fad47dbac8b5f99e17a444f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" is_pre_election: true
I20260923 14:36:19.887418 4479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbca0a9fbd54514a703407b0fdf5df4" candidate_uuid: "c091da166fad47dbac8b5f99e17a444f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a02238e43b544c0874193ee2406ae97" is_pre_election: true
I20260923 14:36:19.887672 4479 raft_consensus.cc:2454] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c091da166fad47dbac8b5f99e17a444f in term 0.
I20260923 14:36:19.887686 4198 raft_consensus.cc:2454] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c091da166fad47dbac8b5f99e17a444f in term 0.
I20260923 14:36:19.888098 4272 leader_election.cc:302] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19fe0fe6f9af40a58e2d9418d5f75244, 9a02238e43b544c0874193ee2406ae97, c091da166fad47dbac8b5f99e17a444f; no voters:
I20260923 14:36:19.888347 4557 raft_consensus.cc:2781] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:19.888428 4557 raft_consensus.cc:482] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:19.888482 4557 raft_consensus.cc:3037] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:19.889473 4557 raft_consensus.cc:504] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } }
I20260923 14:36:19.889802 4557 leader_election.cc:288] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [CANDIDATE]: Term 1 election: Requested vote from peers 9a02238e43b544c0874193ee2406ae97 (127.2.218.3:35669), 19fe0fe6f9af40a58e2d9418d5f75244 (127.2.218.1:44519)
I20260923 14:36:19.890208 4479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbca0a9fbd54514a703407b0fdf5df4" candidate_uuid: "c091da166fad47dbac8b5f99e17a444f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a02238e43b544c0874193ee2406ae97"
I20260923 14:36:19.890326 4479 raft_consensus.cc:3037] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:19.890345 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbca0a9fbd54514a703407b0fdf5df4" candidate_uuid: "c091da166fad47dbac8b5f99e17a444f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fe0fe6f9af40a58e2d9418d5f75244"
I20260923 14:36:19.890508 4198 raft_consensus.cc:3037] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:19.891521 4198 raft_consensus.cc:2454] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c091da166fad47dbac8b5f99e17a444f in term 1.
I20260923 14:36:19.891664 4479 raft_consensus.cc:2454] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c091da166fad47dbac8b5f99e17a444f in term 1.
I20260923 14:36:19.891830 4272 leader_election.cc:302] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fe0fe6f9af40a58e2d9418d5f75244, c091da166fad47dbac8b5f99e17a444f; no voters:
I20260923 14:36:19.892005 4557 raft_consensus.cc:2781] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:19.892215 4557 raft_consensus.cc:686] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 1 LEADER]: Becoming Leader. State: Replica: c091da166fad47dbac8b5f99e17a444f, State: Running, Role: LEADER
I20260923 14:36:19.892513 4557 consensus_queue.cc:237] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } }
I20260923 14:36:19.895184 4039 catalog_manager.cc:5495] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f reported cstate change: term changed from 0 to 1, leader changed from <none> to c091da166fad47dbac8b5f99e17a444f (127.2.218.2). New cstate: current_term: 1 leader_uuid: "c091da166fad47dbac8b5f99e17a444f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } health_report { overall_health: HEALTHY } } }
I20260923 14:36:19.920264 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:19.917049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.2.218.2:45405)
0923 14:36:19.917158 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:19.917161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:19.917174 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:36:19.917336 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:19.917338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:19.917379 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:19.917451 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:19.917654 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.918071 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.918977 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:19.919190 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:19.919301 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:19.919587 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:19.919906 (+ 319us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:19.920014 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:19.920094 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 14:36:19.920145 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:36:19.924893 4479 raft_consensus.cc:1261] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97 [term 1 FOLLOWER]: Refusing update from remote peer c091da166fad47dbac8b5f99e17a444f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:19.924893 4198 raft_consensus.cc:1261] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 FOLLOWER]: Refusing update from remote peer c091da166fad47dbac8b5f99e17a444f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:19.925469 4559 consensus_queue.cc:1035] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:19.925652 4557 consensus_queue.cc:1035] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:19.942360 4040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:19.942500 4040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:19.944679 4040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b: Sending DeleteTablet for 3 replicas of tablet efbca0a9fbd54514a703407b0fdf5df4
I20260923 14:36:19.945333 4459 tablet_service.cc:1514] Processing DeleteTablet for tablet efbca0a9fbd54514a703407b0fdf5df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:19 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:55396
I20260923 14:36:19.945421 4178 tablet_service.cc:1514] Processing DeleteTablet for tablet efbca0a9fbd54514a703407b0fdf5df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:19 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:59074
I20260923 14:36:19.945775 4571 tablet_replica.cc:330] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: stopping tablet replica
I20260923 14:36:19.945811 4318 tablet_service.cc:1514] Processing DeleteTablet for tablet efbca0a9fbd54514a703407b0fdf5df4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:19 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:34892
I20260923 14:36:19.946202 4571 raft_consensus.cc:2229] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:19.946242 4573 tablet_replica.cc:330] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: stopping tablet replica
I20260923 14:36:19.946475 4573 raft_consensus.cc:2229] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:19.946527 4571 raft_consensus.cc:2258] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:19.946663 4040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 14:36:19.946828 4573 raft_consensus.cc:2258] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 14:36:19.947048 4040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:19.948540 4571 ts_tablet_manager.cc:1905] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:19.949071 4572 tablet_replica.cc:330] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: stopping tablet replica
I20260923 14:36:19.949302 4572 raft_consensus.cc:2229] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:19.949527 4572 raft_consensus.cc:2258] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:19.950380 4573 ts_tablet_manager.cc:1905] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:19.951059 4571 ts_tablet_manager.cc:1918] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:19.951160 4571 log.cc:1196] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/efbca0a9fbd54514a703407b0fdf5df4
I20260923 14:36:19.951471 4571 ts_tablet_manager.cc:1939] T efbca0a9fbd54514a703407b0fdf5df4 P 9a02238e43b544c0874193ee2406ae97: Deleting consensus metadata
I20260923 14:36:19.952736 4573 ts_tablet_manager.cc:1918] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:19.952762 4025 catalog_manager.cc:4841] TS 9a02238e43b544c0874193ee2406ae97 (127.2.218.3:35669): tablet efbca0a9fbd54514a703407b0fdf5df4 (table test-table [id=9a1b1afc331f4428afd1b97efeb67612]) successfully deleted
I20260923 14:36:19.952816 4573 log.cc:1196] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/efbca0a9fbd54514a703407b0fdf5df4
I20260923 14:36:19.953080 4573 ts_tablet_manager.cc:1939] T efbca0a9fbd54514a703407b0fdf5df4 P c091da166fad47dbac8b5f99e17a444f: Deleting consensus metadata
I20260923 14:36:19.953899 4459 tablet_service.cc:1467] Processing CreateTablet for tablet 1d54ebd580964e149f0f1b496aad7bb4 (DEFAULT_TABLE table=test-table [id=54c06e0a512845acb3fe6419a6959b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:19.953899 4178 tablet_service.cc:1467] Processing CreateTablet for tablet 1d54ebd580964e149f0f1b496aad7bb4 (DEFAULT_TABLE table=test-table [id=54c06e0a512845acb3fe6419a6959b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:19.954140 4178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d54ebd580964e149f0f1b496aad7bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:19.954146 4459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d54ebd580964e149f0f1b496aad7bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:19.954406 4572 ts_tablet_manager.cc:1905] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:19.956413 4549 tablet_bootstrap.cc:492] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244: Bootstrap starting.
I20260923 14:36:19.956360 4318 tablet_service.cc:1467] Processing CreateTablet for tablet 1d54ebd580964e149f0f1b496aad7bb4 (DEFAULT_TABLE table=test-table [id=54c06e0a512845acb3fe6419a6959b36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:19.956595 4318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d54ebd580964e149f0f1b496aad7bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:19.957038 4572 ts_tablet_manager.cc:1918] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:19.957129 4572 log.cc:1196] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/efbca0a9fbd54514a703407b0fdf5df4
I20260923 14:36:19.957444 4572 ts_tablet_manager.cc:1939] T efbca0a9fbd54514a703407b0fdf5df4 P 19fe0fe6f9af40a58e2d9418d5f75244: Deleting consensus metadata
I20260923 14:36:19.957515 4549 tablet_bootstrap.cc:654] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:19.958232 4550 tablet_bootstrap.cc:492] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: Bootstrap starting.
I20260923 14:36:19.958303 4553 tablet_bootstrap.cc:492] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: Bootstrap starting.
I20260923 14:36:19.959189 4553 tablet_bootstrap.cc:654] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:19.959193 4550 tablet_bootstrap.cc:654] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:19.959446 4027 catalog_manager.cc:4841] TS 19fe0fe6f9af40a58e2d9418d5f75244 (127.2.218.1:44519): tablet efbca0a9fbd54514a703407b0fdf5df4 (table test-table [id=9a1b1afc331f4428afd1b97efeb67612]) successfully deleted
I20260923 14:36:19.959936 4026 catalog_manager.cc:4841] TS c091da166fad47dbac8b5f99e17a444f (127.2.218.2:45405): tablet efbca0a9fbd54514a703407b0fdf5df4 (table test-table [id=9a1b1afc331f4428afd1b97efeb67612]) successfully deleted
I20260923 14:36:19.960531 4553 tablet_bootstrap.cc:492] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: No bootstrap required, opened a new log
I20260923 14:36:19.960608 4553 ts_tablet_manager.cc:1397] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260923 14:36:19.960736 4549 tablet_bootstrap.cc:492] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244: No bootstrap required, opened a new log
I20260923 14:36:19.960827 4549 ts_tablet_manager.cc:1397] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 14:36:19.961061 4553 raft_consensus.cc:348] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:19.961195 4553 raft_consensus.cc:374] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:19.961236 4553 raft_consensus.cc:729] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c091da166fad47dbac8b5f99e17a444f, State: Initialized, Role: FOLLOWER
I20260923 14:36:19.961234 4549 raft_consensus.cc:348] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:19.961357 4549 raft_consensus.cc:374] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:19.961386 4549 raft_consensus.cc:729] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fe0fe6f9af40a58e2d9418d5f75244, State: Initialized, Role: FOLLOWER
I20260923 14:36:19.961375 4553 consensus_queue.cc:260] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:19.961541 4549 consensus_queue.cc:260] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:19.961678 4553 ts_tablet_manager.cc:1428] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260923 14:36:19.961711 4550 tablet_bootstrap.cc:492] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: No bootstrap required, opened a new log
I20260923 14:36:19.961778 4550 ts_tablet_manager.cc:1397] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260923 14:36:19.962227 4550 raft_consensus.cc:348] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:19.962327 4550 raft_consensus.cc:374] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:19.962391 4550 raft_consensus.cc:729] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a02238e43b544c0874193ee2406ae97, State: Initialized, Role: FOLLOWER
I20260923 14:36:19.962517 4550 consensus_queue.cc:260] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:19.962755 4550 ts_tablet_manager.cc:1428] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:19.963689 4549 ts_tablet_manager.cc:1428] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20260923 14:36:20.015710 4527 tablet.cc:2369] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:36:20.047482 4386 tablet.cc:2369] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:20.231822 4556 raft_consensus.cc:482] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:20.231962 4556 raft_consensus.cc:504] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.232973 4556 leader_election.cc:288] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19fe0fe6f9af40a58e2d9418d5f75244 (127.2.218.1:44519), c091da166fad47dbac8b5f99e17a444f (127.2.218.2:45405)
I20260923 14:36:20.237445 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.233330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:51439 (local address 127.2.218.2:45405)
0923 14:36:20.233440 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.233443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.233627 (+ 184us) server_negotiation.cc:407] Connection header received
0923 14:36:20.233720 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.233724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.233769 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.233819 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:20.234062 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.234724 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.236401 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.236914 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.237007 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.237061 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.237273 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.237334 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:36:20.237445 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.233236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:42927 (local address 127.2.218.1:44519)
0923 14:36:20.233368 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.233373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.233389 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:36:20.233526 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.233528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.233566 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.233652 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:20.233933 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.234690 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.236402 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.236914 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.237007 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.237060 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.237273 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.237334 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:36:20.237476 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.233143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:44519 (local address 127.2.218.3:42927)
0923 14:36:20.233310 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:20.233326 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:20.233411 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:20.233680 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:20.233684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:20.233700 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:20.233819 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.233826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.234808 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.234812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.236284 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.236291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.237020 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.237024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.237151 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.237184 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 14:36:20.237244 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 14:36:20.237337 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":263}
I20260923 14:36:20.237491 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.233236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:45405 (local address 127.2.218.3:51439)
0923 14:36:20.233538 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:20.233547 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:20.233643 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:20.233846 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:20.233850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:20.233867 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:20.233969 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.233973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.234808 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.234812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.236281 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.236291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.237027 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.237030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.237137 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.237175 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 14:36:20.237244 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 14:36:20.237340 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":115,"thread_start_us":59,"threads_started":1}
I20260923 14:36:20.237869 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d54ebd580964e149f0f1b496aad7bb4" candidate_uuid: "9a02238e43b544c0874193ee2406ae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" is_pre_election: true
I20260923 14:36:20.238010 4198 raft_consensus.cc:2454] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a02238e43b544c0874193ee2406ae97 in term 0.
I20260923 14:36:20.238157 4339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d54ebd580964e149f0f1b496aad7bb4" candidate_uuid: "9a02238e43b544c0874193ee2406ae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c091da166fad47dbac8b5f99e17a444f" is_pre_election: true
I20260923 14:36:20.238369 4339 raft_consensus.cc:2454] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a02238e43b544c0874193ee2406ae97 in term 0.
I20260923 14:36:20.238343 4413 leader_election.cc:302] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fe0fe6f9af40a58e2d9418d5f75244, 9a02238e43b544c0874193ee2406ae97; no voters:
I20260923 14:36:20.238654 4556 raft_consensus.cc:2781] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:20.238732 4556 raft_consensus.cc:482] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:20.238782 4556 raft_consensus.cc:3037] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:20.239655 4556 raft_consensus.cc:504] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.239970 4556 leader_election.cc:288] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [CANDIDATE]: Term 1 election: Requested vote from peers 19fe0fe6f9af40a58e2d9418d5f75244 (127.2.218.1:44519), c091da166fad47dbac8b5f99e17a444f (127.2.218.2:45405)
I20260923 14:36:20.240263 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d54ebd580964e149f0f1b496aad7bb4" candidate_uuid: "9a02238e43b544c0874193ee2406ae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fe0fe6f9af40a58e2d9418d5f75244"
I20260923 14:36:20.240309 4339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d54ebd580964e149f0f1b496aad7bb4" candidate_uuid: "9a02238e43b544c0874193ee2406ae97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c091da166fad47dbac8b5f99e17a444f"
I20260923 14:36:20.240396 4198 raft_consensus.cc:3037] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:20.240406 4339 raft_consensus.cc:3037] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:20.241164 4198 raft_consensus.cc:2454] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a02238e43b544c0874193ee2406ae97 in term 1.
I20260923 14:36:20.241290 4339 raft_consensus.cc:2454] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a02238e43b544c0874193ee2406ae97 in term 1.
I20260923 14:36:20.241390 4413 leader_election.cc:302] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fe0fe6f9af40a58e2d9418d5f75244, 9a02238e43b544c0874193ee2406ae97; no voters:
I20260923 14:36:20.241549 4556 raft_consensus.cc:2781] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:20.241691 4556 raft_consensus.cc:686] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 1 LEADER]: Becoming Leader. State: Replica: 9a02238e43b544c0874193ee2406ae97, State: Running, Role: LEADER
I20260923 14:36:20.241912 4556 consensus_queue.cc:237] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.243921 4038 catalog_manager.cc:5495] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a02238e43b544c0874193ee2406ae97 (127.2.218.3). New cstate: current_term: 1 leader_uuid: "9a02238e43b544c0874193ee2406ae97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } health_report { overall_health: HEALTHY } } }
Sep 23 14:36:20 dist-test-slave-zl47 krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174178, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:20.346343 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.339317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33117 (local address 127.0.0.1:56628)
0923 14:36:20.339546 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:20.339562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:20.339634 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:20.339831 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:20.339833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:20.340168 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:20.340246 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.340250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.340868 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.340870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.341342 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.341348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.341698 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.341700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.341751 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.342286 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:20.342296 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:20.343496 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:20.344988 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:20.344996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:20.345002 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:20.345202 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:20.345440 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:20.345444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:20.345446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:20.345515 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:20.345827 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:20.345834 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:20.345925 (+ 91us) client_negotiation.cc:769] Sending connection context
0923 14:36:20.346021 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 14:36:20.346175 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260923 14:36:20.346343 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.339411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.2.218.62:33117)
0923 14:36:20.339604 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.339610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.339621 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:36:20.339703 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.339705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.339737 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.339801 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:20.340328 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.340776 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.341434 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.341620 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.341682 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.343620 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:20.343622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:20.343624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:20.343640 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:20.344878 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:20.345287 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:20.345292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:20.345294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:20.345363 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:20.345592 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:20.345595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:20.345597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:20.345737 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:20.345827 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.346069 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.346220 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260923 14:36:20.349318 4038 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:56628:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 14:36:20.350047 4038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:20.354020 4318 tablet_service.cc:1467] Processing CreateTablet for tablet c326ef09bf9e432ba82833c978887eb9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b18436c1c884486b7b66f627f7321da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:36:20.354012 4178 tablet_service.cc:1467] Processing CreateTablet for tablet c326ef09bf9e432ba82833c978887eb9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b18436c1c884486b7b66f627f7321da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:36:20.354213 4459 tablet_service.cc:1467] Processing CreateTablet for tablet c326ef09bf9e432ba82833c978887eb9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b18436c1c884486b7b66f627f7321da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:36:20.354373 4178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c326ef09bf9e432ba82833c978887eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:20.354383 4318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c326ef09bf9e432ba82833c978887eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:20.354583 4459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c326ef09bf9e432ba82833c978887eb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:20.357012 4549 tablet_bootstrap.cc:492] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: Bootstrap starting.
I20260923 14:36:20.357059 4553 tablet_bootstrap.cc:492] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: Bootstrap starting.
I20260923 14:36:20.357887 4553 tablet_bootstrap.cc:654] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:20.357923 4549 tablet_bootstrap.cc:654] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:20.358814 4550 tablet_bootstrap.cc:492] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: Bootstrap starting.
I20260923 14:36:20.359042 4553 tablet_bootstrap.cc:492] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: No bootstrap required, opened a new log
I20260923 14:36:20.359126 4553 ts_tablet_manager.cc:1397] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:20.359500 4553 raft_consensus.cc:348] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.359620 4550 tablet_bootstrap.cc:654] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:20.359615 4553 raft_consensus.cc:374] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:20.359648 4553 raft_consensus.cc:729] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c091da166fad47dbac8b5f99e17a444f, State: Initialized, Role: FOLLOWER
I20260923 14:36:20.359755 4553 consensus_queue.cc:260] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.359998 4553 ts_tablet_manager.cc:1428] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:20.360200 4557 tablet_replica.cc:439] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } }
I20260923 14:36:20.360313 4557 tablet_replica.cc:442] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:20.360852 4549 tablet_bootstrap.cc:492] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: No bootstrap required, opened a new log
I20260923 14:36:20.360919 4549 ts_tablet_manager.cc:1397] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260923 14:36:20.361258 4549 raft_consensus.cc:348] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.361334 4549 raft_consensus.cc:374] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:20.361363 4549 raft_consensus.cc:729] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fe0fe6f9af40a58e2d9418d5f75244, State: Initialized, Role: FOLLOWER
I20260923 14:36:20.361454 4549 consensus_queue.cc:260] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.361600 4550 tablet_bootstrap.cc:492] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: No bootstrap required, opened a new log
I20260923 14:36:20.361644 4549 ts_tablet_manager.cc:1428] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 14:36:20.361676 4550 ts_tablet_manager.cc:1397] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 14:36:20.361708 4554 tablet_replica.cc:439] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } }
I20260923 14:36:20.361831 4554 tablet_replica.cc:442] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:20.362058 4550 raft_consensus.cc:348] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.362182 4550 raft_consensus.cc:374] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:20.362228 4550 raft_consensus.cc:729] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a02238e43b544c0874193ee2406ae97, State: Initialized, Role: FOLLOWER
I20260923 14:36:20.362350 4550 consensus_queue.cc:260] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.362562 4550 ts_tablet_manager.cc:1428] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:20.362752 4556 tablet_replica.cc:439] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } }
I20260923 14:36:20.362851 4556 tablet_replica.cc:442] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:20.460214 4554 raft_consensus.cc:482] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:20.460338 4554 raft_consensus.cc:504] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.461270 4554 leader_election.cc:288] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c091da166fad47dbac8b5f99e17a444f (127.2.218.2:45405), 9a02238e43b544c0874193ee2406ae97 (127.2.218.3:35669)
I20260923 14:36:20.465039 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.461615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:57241 (local address 127.2.218.3:35669)
0923 14:36:20.461819 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.461825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.461841 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:36:20.461994 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.461997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.462048 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.462217 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:20.462458 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.462924 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.463850 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.464442 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.464530 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.464593 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.464816 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.464921 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260923 14:36:20.465039 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.461519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:35669 (local address 127.2.218.1:57241)
0923 14:36:20.461775 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:20.461786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:20.461860 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:20.462230 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:20.462233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:20.462249 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:20.462358 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.462363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.463025 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.463030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.463743 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.463751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.464550 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.464552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.464661 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.464714 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 14:36:20.464793 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 14:36:20.464884 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":18,"thread_start_us":64,"threads_started":1}
I20260923 14:36:20.465039 4592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.461519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:45405 (local address 127.2.218.1:50509)
0923 14:36:20.461994 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:20.462004 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:20.462214 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:20.462459 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:20.462462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:20.462476 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:20.462580 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.462585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.463229 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.463234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.463855 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.463863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.464667 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.464670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.464796 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.464813 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:36:20.464887 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 14:36:20.464930 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":209,"thread_start_us":52,"threads_started":1}
I20260923 14:36:20.465098 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.461591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:50509 (local address 127.2.218.2:45405)
0923 14:36:20.461709 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.461714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.462168 (+ 454us) server_negotiation.cc:407] Connection header received
0923 14:36:20.462331 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.462333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.462372 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.462441 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:20.462678 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.463120 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.464001 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.464553 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.464668 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.464705 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.464912 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.464996 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:36:20.465525 4339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c326ef09bf9e432ba82833c978887eb9" candidate_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c091da166fad47dbac8b5f99e17a444f" is_pre_election: true
I20260923 14:36:20.465540 4479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c326ef09bf9e432ba82833c978887eb9" candidate_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a02238e43b544c0874193ee2406ae97" is_pre_election: true
I20260923 14:36:20.465663 4339 raft_consensus.cc:2454] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fe0fe6f9af40a58e2d9418d5f75244 in term 0.
I20260923 14:36:20.465663 4479 raft_consensus.cc:2454] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19fe0fe6f9af40a58e2d9418d5f75244 in term 0.
I20260923 14:36:20.465961 4131 leader_election.cc:302] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fe0fe6f9af40a58e2d9418d5f75244, c091da166fad47dbac8b5f99e17a444f; no voters:
I20260923 14:36:20.466234 4554 raft_consensus.cc:2781] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:20.466298 4554 raft_consensus.cc:482] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:20.466347 4554 raft_consensus.cc:3037] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:20.467245 4554 raft_consensus.cc:504] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.467598 4554 leader_election.cc:288] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [CANDIDATE]: Term 1 election: Requested vote from peers c091da166fad47dbac8b5f99e17a444f (127.2.218.2:45405), 9a02238e43b544c0874193ee2406ae97 (127.2.218.3:35669)
I20260923 14:36:20.467907 4339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c326ef09bf9e432ba82833c978887eb9" candidate_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c091da166fad47dbac8b5f99e17a444f"
I20260923 14:36:20.467993 4339 raft_consensus.cc:3037] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:20.467916 4479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c326ef09bf9e432ba82833c978887eb9" candidate_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a02238e43b544c0874193ee2406ae97"
I20260923 14:36:20.468057 4479 raft_consensus.cc:3037] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:20.468813 4479 raft_consensus.cc:2454] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fe0fe6f9af40a58e2d9418d5f75244 in term 1.
I20260923 14:36:20.468894 4339 raft_consensus.cc:2454] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19fe0fe6f9af40a58e2d9418d5f75244 in term 1.
I20260923 14:36:20.469110 4130 leader_election.cc:302] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fe0fe6f9af40a58e2d9418d5f75244, 9a02238e43b544c0874193ee2406ae97; no voters:
I20260923 14:36:20.469286 4554 raft_consensus.cc:2781] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:20.469528 4554 raft_consensus.cc:686] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 LEADER]: Becoming Leader. State: Replica: 19fe0fe6f9af40a58e2d9418d5f75244, State: Running, Role: LEADER
I20260923 14:36:20.469771 4554 consensus_queue.cc:237] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } }
I20260923 14:36:20.470890 4593 tablet_replica.cc:439] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: TxnStatusTablet state changed. Reason: New leader 19fe0fe6f9af40a58e2d9418d5f75244. Latest consensus state: current_term: 1 leader_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } }
I20260923 14:36:20.471009 4593 tablet_replica.cc:442] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:36:20.471252 4595 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 14:36:20.471390 4595 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 14:36:20.471990 4038 catalog_manager.cc:5495] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19fe0fe6f9af40a58e2d9418d5f75244 (127.2.218.1). New cstate: current_term: 1 leader_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } health_report { overall_health: UNKNOWN } } }
I20260923 14:36:20.510716 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.507609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.2.218.1:44519)
0923 14:36:20.507721 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.507725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.507737 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:20.507865 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.507868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.507922 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.508001 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:20.508237 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.508670 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.509402 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.509615 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.509710 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.509983 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:20.510383 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:20.510486 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.510544 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.510598 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:36:20.510717 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.507510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:44519 (local address 127.0.0.1:59086)
0923 14:36:20.507621 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:20.507636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:20.507725 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:20.508025 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:20.508027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:20.508035 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:20.508136 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.508140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.508765 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.508769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.509293 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.509300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.509709 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.509712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.509816 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.509853 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:20.510463 (+ 610us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:20.510477 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 14:36:20.510537 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 14:36:20.510615 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 14:36:20.513545 4479 raft_consensus.cc:1261] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97 [term 1 FOLLOWER]: Refusing update from remote peer 19fe0fe6f9af40a58e2d9418d5f75244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:20.513676 4339 raft_consensus.cc:1261] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f [term 1 FOLLOWER]: Refusing update from remote peer 19fe0fe6f9af40a58e2d9418d5f75244: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:20.513936 4554 consensus_queue.cc:1035] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:20.514123 4593 consensus_queue.cc:1035] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:20.515492 4556 tablet_replica.cc:439] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: TxnStatusTablet state changed. Reason: New leader 19fe0fe6f9af40a58e2d9418d5f75244. Latest consensus state: current_term: 1 leader_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } }
I20260923 14:36:20.515597 4556 tablet_replica.cc:442] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:20.515697 4557 tablet_replica.cc:439] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: TxnStatusTablet state changed. Reason: New leader 19fe0fe6f9af40a58e2d9418d5f75244. Latest consensus state: current_term: 1 leader_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } }
I20260923 14:36:20.515782 4557 tablet_replica.cc:442] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:20.517711 4598 mvcc.cc:205] Tried to move back new op lower bound from 7332553443379224576 to 7332553443206492160. Current Snapshot: MvccSnapshot[applied={T|T < 7332553443379224576}]
I20260923 14:36:20.517796 4554 tablet_replica.cc:439] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } }
I20260923 14:36:20.517899 4554 tablet_replica.cc:442] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:36:20.518304 4557 tablet_replica.cc:439] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } }
I20260923 14:36:20.518394 4557 tablet_replica.cc:442] T c326ef09bf9e432ba82833c978887eb9 P c091da166fad47dbac8b5f99e17a444f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:20.519832 4600 mvcc.cc:205] Tried to move back new op lower bound from 7332553443379224576 to 7332553443206492160. Current Snapshot: MvccSnapshot[applied={T|T < 7332553443379224576}]
I20260923 14:36:20.520051 4593 tablet_replica.cc:439] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } }
I20260923 14:36:20.520150 4593 tablet_replica.cc:442] T c326ef09bf9e432ba82833c978887eb9 P 19fe0fe6f9af40a58e2d9418d5f75244: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:36:20.521044 4556 tablet_replica.cc:439] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 } } peers { permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 } } peers { permanent_uuid: "9a02238e43b544c0874193ee2406ae97" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 35669 } } }
I20260923 14:36:20.521149 4556 tablet_replica.cc:442] T c326ef09bf9e432ba82833c978887eb9 P 9a02238e43b544c0874193ee2406ae97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:20.526616 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.523224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.2.218.3:35669)
0923 14:36:20.523319 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.523322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.523471 (+ 149us) server_negotiation.cc:407] Connection header received
0923 14:36:20.523568 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.523570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.523621 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.523713 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:20.523852 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.524498 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.525185 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.525474 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.525566 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.525818 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:20.526296 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:20.526393 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.526449 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.526495 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:36:20.534925 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.532070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:41833 (local address 127.2.218.1:44519)
0923 14:36:20.532160 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.532163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.532283 (+ 120us) server_negotiation.cc:407] Connection header received
0923 14:36:20.532377 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.532379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.532408 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.532458 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:20.532678 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.533059 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.534079 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.534305 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.534371 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.534628 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:20.534678 (+ 50us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:20.534722 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.534814 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.534846 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:36:20.534996 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.531951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:44519 (local address 127.2.218.3:41833)
0923 14:36:20.532204 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:20.532216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:20.532310 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:20.532489 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:20.532492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:20.532498 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:20.532592 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.532597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.533159 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.533162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.533972 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.533981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.534402 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.534405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.534484 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.534523 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:20.534751 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:20.534756 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:36:20.534813 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 14:36:20.534882 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260923 14:36:20.540529 4339 raft_consensus.cc:1261] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f [term 1 FOLLOWER]: Refusing update from remote peer 9a02238e43b544c0874193ee2406ae97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:20.540606 4198 raft_consensus.cc:1261] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 FOLLOWER]: Refusing update from remote peer 9a02238e43b544c0874193ee2406ae97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:20.540935 4578 consensus_queue.cc:1035] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c091da166fad47dbac8b5f99e17a444f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:20.541096 4556 consensus_queue.cc:1035] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fe0fe6f9af40a58e2d9418d5f75244" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 44519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:20.547020 4599 mvcc.cc:205] Tried to move back new op lower bound from 7332553443490439168 to 7332553442273247232. Current Snapshot: MvccSnapshot[applied={T|T < 7332553443490439168}]
I20260923 14:36:20.564464 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.560903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:58939 (local address 127.2.218.3:35669)
0923 14:36:20.561005 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.561009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.561191 (+ 182us) server_negotiation.cc:407] Connection header received
0923 14:36:20.561309 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.561311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.561348 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.561424 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:20.561683 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.562361 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.563443 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.563741 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.563799 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.564106 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:20.564172 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:20.564208 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.564330 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.564368 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:36:20.564742 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.560596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:35669 (local address 127.2.218.1:58939)
0923 14:36:20.561111 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:20.561127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:20.561227 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:20.561507 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:20.561510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:20.561516 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:20.561595 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.561600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.562477 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.562479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.563333 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:20.563342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.563888 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.563890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:20.563954 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.563989 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:20.564258 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:20.564262 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:36:20.564564 (+ 302us) client_negotiation.cc:240] Negotiation successful
0923 14:36:20.564627 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":78,"threads_started":1}
I20260923 14:36:20.584954 4040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:20.585100 4040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:20.586988 4040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b30290f65ac04e11a88bb60a254f6b1b: Sending DeleteTablet for 3 replicas of tablet 1d54ebd580964e149f0f1b496aad7bb4
I20260923 14:36:20.587394 4178 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d54ebd580964e149f0f1b496aad7bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:20 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:59074
I20260923 14:36:20.587694 4620 tablet_replica.cc:330] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244: stopping tablet replica
I20260923 14:36:20.587805 4620 raft_consensus.cc:2229] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:20.587832 4459 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d54ebd580964e149f0f1b496aad7bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:20 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:55396
I20260923 14:36:20.587888 4620 raft_consensus.cc:2258] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:20.588420 4620 ts_tablet_manager.cc:1905] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:20.589184 4318 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d54ebd580964e149f0f1b496aad7bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:20 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:34892
I20260923 14:36:20.591089 4620 ts_tablet_manager.cc:1918] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:20.591163 4620 log.cc:1196] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/1d54ebd580964e149f0f1b496aad7bb4
I20260923 14:36:20.591413 4620 ts_tablet_manager.cc:1939] T 1d54ebd580964e149f0f1b496aad7bb4 P 19fe0fe6f9af40a58e2d9418d5f75244: Deleting consensus metadata
I20260923 14:36:20.591950 4027 catalog_manager.cc:4841] TS 19fe0fe6f9af40a58e2d9418d5f75244 (127.2.218.1:44519): tablet 1d54ebd580964e149f0f1b496aad7bb4 (table test-table [id=54c06e0a512845acb3fe6419a6959b36]) successfully deleted
I20260923 14:36:20.592126 4621 tablet_replica.cc:330] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: stopping tablet replica
I20260923 14:36:20.592233 4621 raft_consensus.cc:2229] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:20.592458 4621 raft_consensus.cc:2258] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:20.593011 4621 ts_tablet_manager.cc:1905] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:20.595329 4621 ts_tablet_manager.cc:1918] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:20.595396 4621 log.cc:1196] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/1d54ebd580964e149f0f1b496aad7bb4
I20260923 14:36:20.595614 4621 ts_tablet_manager.cc:1939] T 1d54ebd580964e149f0f1b496aad7bb4 P 9a02238e43b544c0874193ee2406ae97: Deleting consensus metadata
I20260923 14:36:20.596099 4025 catalog_manager.cc:4841] TS 9a02238e43b544c0874193ee2406ae97 (127.2.218.3:35669): tablet 1d54ebd580964e149f0f1b496aad7bb4 (table test-table [id=54c06e0a512845acb3fe6419a6959b36]) successfully deleted
I20260923 14:36:20.596344 4623 tablet_replica.cc:330] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: stopping tablet replica
I20260923 14:36:20.596465 4623 raft_consensus.cc:2229] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:20.596549 4623 raft_consensus.cc:2258] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 14:36:20 dist-test-slave-zl47 krb5kdc[3995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:36:20.597497 4623 ts_tablet_manager.cc:1905] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:20.599782 4623 ts_tablet_manager.cc:1918] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:20.599852 4623 log.cc:1196] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/1d54ebd580964e149f0f1b496aad7bb4
I20260923 14:36:20.600081 4623 ts_tablet_manager.cc:1939] T 1d54ebd580964e149f0f1b496aad7bb4 P c091da166fad47dbac8b5f99e17a444f: Deleting consensus metadata
I20260923 14:36:20.600590 4026 catalog_manager.cc:4841] TS c091da166fad47dbac8b5f99e17a444f (127.2.218.2:45405): tablet 1d54ebd580964e149f0f1b496aad7bb4 (table test-table [id=54c06e0a512845acb3fe6419a6959b36]) successfully deleted
I20260923 14:36:20.601783 4547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.589700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.2.218.1:44519)
0923 14:36:20.589786 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.589790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.590250 (+ 460us) server_negotiation.cc:407] Connection header received
0923 14:36:20.590407 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.590410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.590449 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.590495 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:20.591450 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.592072 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.593336 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.593629 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.593701 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.597806 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:20.597815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:20.597826 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:20.597847 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:20.599667 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:20.600463 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:20.600469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:20.600470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:20.600555 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:20.600939 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:20.600942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:20.600944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:20.601176 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:20.601347 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.601556 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.601658 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260923 14:36:20.602217 4238 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59096
I20260923 14:36:20.611478 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:20.603595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.2.218.62:33117)
0923 14:36:20.603702 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:20.603708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:20.603979 (+ 271us) server_negotiation.cc:407] Connection header received
0923 14:36:20.604124 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:20.604128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:20.604174 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:20.604213 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:20.604906 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.605579 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.606525 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:20.606812 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:20.606877 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:20.608261 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:20.608265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:20.608267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:20.608289 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:20.609871 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:20.610325 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:20.610328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:20.610330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:20.610407 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:20.610675 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:20.610678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:20.610680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:20.610857 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:20.610955 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:20.611243 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 14:36:20.611344 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260923 14:36:20.612376 4040 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56644
I20260923 14:36:20.613415 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 4113
I20260923 14:36:20.621977 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 4253
I20260923 14:36:20.628074 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 4394
I20260923 14:36:20.632511 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 4008
2026-09-23T14:36:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2780 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:20 dist-test-slave-zl47 krb5kdc[4629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:20 dist-test-slave-zl47 krb5kdc[4629](info): set up 2 sockets
Sep 23 14:36:20 dist-test-slave-zl47 krb5kdc[4629](info): commencing operation
krb5kdc: starting...
W20260923 14:36:21.529937 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:21 dist-test-slave-zl47 krb5kdc[4629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:21.626574 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:45511
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:45511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:21.720368 4642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:21.720592 4642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:21.720654 4642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:21.723985 4642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:21.724053 4642 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 14:36:21.724081 4642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:21.724115 4642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:21.724150 4642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:21.724180 4642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:21.728358 4642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:45511
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:45511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:21.729113 4642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:21.729899 4642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:36:21.736923 4642 server_base.cc:1034] running on GCE node
W20260923 14:36:21.736863 4648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:21.736866 4650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:21.736871 4647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:21.737535 4642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:21.738474 4642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:21.739631 4642 hybrid_clock.cc:648] HybridClock initialized: now 1790174181739617 us; error 27 us; skew 500 ppm
Sep 23 14:36:21 dist-test-slave-zl47 krb5kdc[4629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174181, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:21.741868 4642 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:21.742609 4642 webserver.cc:460] Webserver started at http://127.2.218.62:43475/ using document root <none> and password file <none>
I20260923 14:36:21.743062 4642 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:21.743137 4642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:21.743327 4642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:21.744926 4642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "66855d46195b4de294ff9caaf1ac7cab"
format_stamp: "Formatted at 2026-09-23 14:36:21 on dist-test-slave-zl47"
server_key: "1c8e163c0b5b18e41cc9243f3747934f"
server_key_iv: "365a2e8d04e38bb134960c0b0ec401ff"
server_key_version: "encryptionkey@0"
I20260923 14:36:21.745391 4642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "66855d46195b4de294ff9caaf1ac7cab"
format_stamp: "Formatted at 2026-09-23 14:36:21 on dist-test-slave-zl47"
server_key: "1c8e163c0b5b18e41cc9243f3747934f"
server_key_iv: "365a2e8d04e38bb134960c0b0ec401ff"
server_key_version: "encryptionkey@0"
I20260923 14:36:21.748355 4642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:21.750267 4657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:21.751250 4642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:21.751370 4642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "66855d46195b4de294ff9caaf1ac7cab"
format_stamp: "Formatted at 2026-09-23 14:36:21 on dist-test-slave-zl47"
server_key: "1c8e163c0b5b18e41cc9243f3747934f"
server_key_iv: "365a2e8d04e38bb134960c0b0ec401ff"
server_key_version: "encryptionkey@0"
I20260923 14:36:21.751463 4642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:21.770620 4642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:21.772747 4642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:21.772913 4642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:21.779671 4642 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:45511
I20260923 14:36:21.779696 4720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:45511 every 8 connection(s)
I20260923 14:36:21.780421 4642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:21.780854 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 4642
I20260923 14:36:21.780965 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:21.781209 2920 external_mini_cluster.cc:1428] Setting key 36a43c16217132ce36e30e151d6db965
I20260923 14:36:21.782996 4721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:21.788664 4721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab: Bootstrap starting.
Sep 23 14:36:21 dist-test-slave-zl47 krb5kdc[4629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:21.790819 4721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:21.791791 4721 log.cc:824] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:21.793706 4721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab: No bootstrap required, opened a new log
I20260923 14:36:21.794281 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:21.782376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.2.218.62:45511)
0923 14:36:21.782754 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:21.782767 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:21.782795 (+ 28us) server_negotiation.cc:407] Connection header received
0923 14:36:21.783408 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:21.783424 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:21.783699 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:21.783974 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:21.784812 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:21.785728 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:21.786883 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:21.787266 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:21.787366 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:21.789850 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:21.789854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:21.789864 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:21.789887 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:21.791571 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:21.792023 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:21.792028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:21.792033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:21.792168 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:21.792454 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:21.792456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:21.792457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:21.792894 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:21.793054 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:21.793278 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 14:36:21.793507 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":126,"threads_started":1}
I20260923 14:36:21.796286 4721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66855d46195b4de294ff9caaf1ac7cab" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45511 } }
I20260923 14:36:21.796458 4721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:21.796487 4721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66855d46195b4de294ff9caaf1ac7cab, State: Initialized, Role: FOLLOWER
I20260923 14:36:21.796774 4721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66855d46195b4de294ff9caaf1ac7cab" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45511 } }
I20260923 14:36:21.796875 4721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:21.796911 4721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:21.796965 4721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:21.797588 4721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66855d46195b4de294ff9caaf1ac7cab" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45511 } }
I20260923 14:36:21.797822 4721 leader_election.cc:302] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66855d46195b4de294ff9caaf1ac7cab; no voters:
I20260923 14:36:21.798020 4721 leader_election.cc:288] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:21.798149 4726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:21.798377 4726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [term 1 LEADER]: Becoming Leader. State: Replica: 66855d46195b4de294ff9caaf1ac7cab, State: Running, Role: LEADER
I20260923 14:36:21.798671 4726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66855d46195b4de294ff9caaf1ac7cab" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45511 } }
I20260923 14:36:21.799041 4721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:21.800063 4727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66855d46195b4de294ff9caaf1ac7cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66855d46195b4de294ff9caaf1ac7cab" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45511 } } }
I20260923 14:36:21.800204 4727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:21.801213 4728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66855d46195b4de294ff9caaf1ac7cab. Latest consensus state: current_term: 1 leader_uuid: "66855d46195b4de294ff9caaf1ac7cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66855d46195b4de294ff9caaf1ac7cab" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45511 } } }
I20260923 14:36:21.801314 4728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab [sys.catalog]: This master's current role is: LEADER
W20260923 14:36:21.802995 4741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:36:21.803090 4741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:36:21.803171 4737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:21.804070 4737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:21.809131 4737 catalog_manager.cc:1347] Generated new cluster ID: 4fa54702c641495b9d041638e44a3d65
I20260923 14:36:21.809201 4737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:21.816649 4737 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:21.817667 4737 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:21.828642 4737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab: Generated new TSK 0
I20260923 14:36:21.829322 4737 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:36:21.856452 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:45511
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:21.951331 4747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:21.951579 4747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:21.951656 4747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:21.955509 4747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:21.955565 4747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:21.955636 4747 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 14:36:21.955675 4747 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:36:21.955746 4747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:21.960129 4747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.218.62:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:21.960984 4747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:21.961889 4747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:21.970052 4753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:21.970116 4752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:21.970065 4755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:21.970427 4747 server_base.cc:1034] running on GCE node
I20260923 14:36:21.970803 4747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:21.971356 4747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:21.972497 4747 hybrid_clock.cc:648] HybridClock initialized: now 1790174181972475 us; error 36 us; skew 500 ppm
Sep 23 14:36:21 dist-test-slave-zl47 krb5kdc[4629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174181, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:21.975067 4747 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:36:21.975832 4747 webserver.cc:460] Webserver started at http://127.2.218.1:42069/ using document root <none> and password file <none>
I20260923 14:36:21.976312 4747 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:21.976377 4747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:21.976573 4747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:21.978308 4747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "a751bdab1bed4d528853db392edee82a"
format_stamp: "Formatted at 2026-09-23 14:36:21 on dist-test-slave-zl47"
server_key: "6e186aff3f7d06a189e6121265dbd23d"
server_key_iv: "971095a9b743ed57004a163dada87af3"
server_key_version: "encryptionkey@0"
I20260923 14:36:21.978777 4747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "a751bdab1bed4d528853db392edee82a"
format_stamp: "Formatted at 2026-09-23 14:36:21 on dist-test-slave-zl47"
server_key: "6e186aff3f7d06a189e6121265dbd23d"
server_key_iv: "971095a9b743ed57004a163dada87af3"
server_key_version: "encryptionkey@0"
I20260923 14:36:21.981923 4747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:36:21.984053 4762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:21.985045 4747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:21.985154 4747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "a751bdab1bed4d528853db392edee82a"
format_stamp: "Formatted at 2026-09-23 14:36:21 on dist-test-slave-zl47"
server_key: "6e186aff3f7d06a189e6121265dbd23d"
server_key_iv: "971095a9b743ed57004a163dada87af3"
server_key_version: "encryptionkey@0"
I20260923 14:36:21.985237 4747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:22.010550 4747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:22.012804 4747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:22.012995 4747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:22.014619 4747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:22.014671 4747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:22.014710 4747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:22.014765 4747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:36:22 dist-test-slave-zl47 krb5kdc[4629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174181, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:22.026875 4747 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:46373
I20260923 14:36:22.026901 4878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:46373 every 8 connection(s)
I20260923 14:36:22.027786 4747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:22.028249 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.015456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:45883 (local address 127.2.218.62:45511)
0923 14:36:22.015618 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.015624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.016461 (+ 837us) server_negotiation.cc:407] Connection header received
0923 14:36:22.017761 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.017766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.017875 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.017978 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:22.019537 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.020052 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.021631 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.021987 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.022115 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.024541 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:22.024544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:22.024546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:22.024570 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:22.026261 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.026775 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.026778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.026780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.026867 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.027221 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.027224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.027225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.027386 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:22.027486 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.027918 (+ 432us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.028075 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:36:22.028959 4771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.015841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45511 (local address 127.2.218.1:45883)
0923 14:36:22.016295 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.016345 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.017507 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.018211 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.018222 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.018751 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:22.019378 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.019399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.020176 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.020180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.021503 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.021517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.022136 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.022140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.022299 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.022825 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:22.022840 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:22.024369 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:22.026388 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.026394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.026404 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.026690 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.027014 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.027017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.027019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.027132 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.027497 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:22.027504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:22.027707 (+ 203us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.027858 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.028159 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":100,"threads_started":1}
I20260923 14:36:22.030155 4879 heartbeater.cc:347] Connected to a master server at 127.2.218.62:45511
I20260923 14:36:22.030304 4879 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:22.030769 4879 heartbeater.cc:511] Master 127.2.218.62:45511 requested a full tablet report, sending...
I20260923 14:36:22.031744 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 4747
I20260923 14:36:22.031886 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:22.032226 2920 external_mini_cluster.cc:1428] Setting key 443240d515572c8ba3cc38384ff1f817
I20260923 14:36:22.032238 4674 ts_manager.cc:194] Registered new tserver with Master: a751bdab1bed4d528853db392edee82a (127.2.218.1:46373)
I20260923 14:36:22.033633 4674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:45883
I20260923 14:36:22.041605 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.032607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:55405 (local address 127.2.218.62:45511)
0923 14:36:22.032728 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.032732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.032745 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:36:22.032775 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.032777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.032811 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.032885 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:22.034175 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.034833 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.036023 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.036371 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.036487 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.037583 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:22.037587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:22.037589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:22.037608 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:22.039778 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.040317 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.040322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.040326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.040431 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.040742 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.040748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.040750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.040966 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:22.041066 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.041357 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.041457 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1185,"server-negotiator.queue_time_us":54}
I20260923 14:36:22.041612 4885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.031888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45511 (local address 127.2.218.1:55405)
0923 14:36:22.032160 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.032174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.032254 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.032979 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.032982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.033163 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:22.033252 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.033257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.034997 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.035000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.035889 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.035903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.036485 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.036487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.036581 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.037039 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:22.037071 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:22.037452 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:22.039888 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.039893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.039897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.040163 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.040532 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.040537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.040541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.040638 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.041114 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:22.041117 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:22.041198 (+ 81us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.041307 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.041429 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:36:22.061151 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:45511
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:22.153029 4888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:22.153227 4888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:22.153303 4888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:22.156723 4888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:22.156766 4888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:22.156827 4888 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 14:36:22.156855 4888 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:36:22.156920 4888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:22.161481 4888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.218.62:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:22.162288 4888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:22.163028 4888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:22.170831 4894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:22.170840 4896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:22.170876 4893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:22.171010 4888 server_base.cc:1034] running on GCE node
I20260923 14:36:22.171350 4888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:22.171864 4888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:22.173091 4888 hybrid_clock.cc:648] HybridClock initialized: now 1790174182173081 us; error 30 us; skew 500 ppm
Sep 23 14:36:22 dist-test-slave-zl47 krb5kdc[4629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174182, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:22.175508 4888 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:36:22.176148 4888 webserver.cc:460] Webserver started at http://127.2.218.2:46673/ using document root <none> and password file <none>
I20260923 14:36:22.176571 4888 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:22.176654 4888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:22.176838 4888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:22.178422 4888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "14b7dfcaf4264ecdba46080a6a4fec12"
format_stamp: "Formatted at 2026-09-23 14:36:22 on dist-test-slave-zl47"
server_key: "bd6ab31b5a0271098d8d42c31ba5143e"
server_key_iv: "0e037b39fe5fd15fed2a388030109777"
server_key_version: "encryptionkey@0"
I20260923 14:36:22.178901 4888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "14b7dfcaf4264ecdba46080a6a4fec12"
format_stamp: "Formatted at 2026-09-23 14:36:22 on dist-test-slave-zl47"
server_key: "bd6ab31b5a0271098d8d42c31ba5143e"
server_key_iv: "0e037b39fe5fd15fed2a388030109777"
server_key_version: "encryptionkey@0"
I20260923 14:36:22.181895 4888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:36:22.183995 4903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:22.184967 4888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:22.185101 4888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "14b7dfcaf4264ecdba46080a6a4fec12"
format_stamp: "Formatted at 2026-09-23 14:36:22 on dist-test-slave-zl47"
server_key: "bd6ab31b5a0271098d8d42c31ba5143e"
server_key_iv: "0e037b39fe5fd15fed2a388030109777"
server_key_version: "encryptionkey@0"
I20260923 14:36:22.185233 4888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:22.195959 4888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:22.198019 4888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:22.198217 4888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:22.199318 4888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:22.199380 4888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:22.199432 4888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:22.199460 4888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:36:22 dist-test-slave-zl47 krb5kdc[4629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174182, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:22.208729 4888 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:34535
I20260923 14:36:22.208782 5019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:34535 every 8 connection(s)
I20260923 14:36:22.209544 4888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:36:22.210697 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.199979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:40533 (local address 127.2.218.62:45511)
0923 14:36:22.200124 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.200130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.201118 (+ 988us) server_negotiation.cc:407] Connection header received
0923 14:36:22.202201 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.202204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.202262 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.202356 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:22.203396 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.203895 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.204935 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.205182 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.205238 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.207500 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:22.207502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:22.207504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:22.207524 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:22.208910 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.209308 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.209312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.209313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.209390 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.209693 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.209696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.209698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.209854 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:22.209946 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.210446 (+ 500us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.210552 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:36:22.211274 4912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.200626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45511 (local address 127.2.218.2:40533)
0923 14:36:22.200991 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.201029 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.201975 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.202492 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.202505 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.202844 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:22.203267 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.203278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.204006 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.204009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.204789 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.204797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.205401 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.205404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.205541 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.206034 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:22.206046 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:22.207356 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:22.208999 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.209003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.209011 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.209214 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.209470 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.209472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.209473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.209606 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.209955 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:22.209962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:22.210242 (+ 280us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.210406 (+ 164us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.210656 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":116,"threads_started":1}
I20260923 14:36:22.212208 5020 heartbeater.cc:347] Connected to a master server at 127.2.218.62:45511
I20260923 14:36:22.212317 5020 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:22.212781 5020 heartbeater.cc:511] Master 127.2.218.62:45511 requested a full tablet report, sending...
I20260923 14:36:22.213742 4673 ts_manager.cc:194] Registered new tserver with Master: 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535)
I20260923 14:36:22.214430 4673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:40533
I20260923 14:36:22.215673 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 4888
I20260923 14:36:22.215787 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:22.216001 2920 external_mini_cluster.cc:1428] Setting key 9740993170285b23a7a768e9318f3e14
I20260923 14:36:22.220332 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.214204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:47473 (local address 127.2.218.62:45511)
0923 14:36:22.214287 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.214290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.214411 (+ 121us) server_negotiation.cc:407] Connection header received
0923 14:36:22.214470 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.214472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.214510 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.214576 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:22.215047 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.215481 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.216312 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.216541 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.216639 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.217523 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:22.217528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:22.217533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:22.217557 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:22.218956 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.219381 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.219383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.219384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.219442 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.219691 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.219693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.219694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.219851 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:22.219915 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.220176 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.220247 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 14:36:22.220492 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.214098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45511 (local address 127.2.218.2:47473)
0923 14:36:22.214324 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.214340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.214417 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.214685 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.214687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.214814 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:22.214938 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.214944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.215609 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.215612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.216211 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.216219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.216655 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.216658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.216737 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.217130 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:22.217141 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:22.217407 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:22.219071 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.219075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.219078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.219298 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.219518 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.219522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.219524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.219617 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.219935 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:22.219941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:22.220033 (+ 92us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.220133 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.220267 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:36:22.239051 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:45511
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:22.329969 5029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:22.330200 5029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:22.330240 5029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:22.333509 5029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:22.333550 5029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:22.333587 5029 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 14:36:22.333599 5029 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:36:22.333626 5029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:22.337745 5029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.218.62:45511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:22.338452 5029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:22.339184 5029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:22.346529 5037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:22.346691 5029 server_base.cc:1034] running on GCE node
W20260923 14:36:22.346529 5034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:22.346545 5035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:22.347160 5029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:22.347580 5029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:22.348778 5029 hybrid_clock.cc:648] HybridClock initialized: now 1790174182348755 us; error 32 us; skew 500 ppm
Sep 23 14:36:22 dist-test-slave-zl47 krb5kdc[4629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174182, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:22.351079 5029 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:36:22.351754 5029 webserver.cc:460] Webserver started at http://127.2.218.3:33475/ using document root <none> and password file <none>
I20260923 14:36:22.352185 5029 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:22.352259 5029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:22.352439 5029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:22.354058 5029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "613a5aaa219641a683e0ad1d5005a4b0"
format_stamp: "Formatted at 2026-09-23 14:36:22 on dist-test-slave-zl47"
server_key: "a1dda0176810057ae226eb37fd1d4c7b"
server_key_iv: "06c4f92506262be11321d9ac577868dd"
server_key_version: "encryptionkey@0"
I20260923 14:36:22.354641 5029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "613a5aaa219641a683e0ad1d5005a4b0"
format_stamp: "Formatted at 2026-09-23 14:36:22 on dist-test-slave-zl47"
server_key: "a1dda0176810057ae226eb37fd1d4c7b"
server_key_iv: "06c4f92506262be11321d9ac577868dd"
server_key_version: "encryptionkey@0"
I20260923 14:36:22.357437 5029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:36:22.359320 5044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:22.360213 5029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:22.360324 5029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "613a5aaa219641a683e0ad1d5005a4b0"
format_stamp: "Formatted at 2026-09-23 14:36:22 on dist-test-slave-zl47"
server_key: "a1dda0176810057ae226eb37fd1d4c7b"
server_key_iv: "06c4f92506262be11321d9ac577868dd"
server_key_version: "encryptionkey@0"
I20260923 14:36:22.360410 5029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:22.382467 5029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:22.384831 5029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:22.384989 5029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:22.386152 5029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:22.386227 5029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:22.386282 5029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:22.386328 5029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:36:22 dist-test-slave-zl47 krb5kdc[4629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174182, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:22.395462 5029 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:34785
I20260923 14:36:22.395506 5159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:34785 every 8 connection(s)
I20260923 14:36:22.396360 5029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:36:22.398445 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.386648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:59115 (local address 127.2.218.62:45511)
0923 14:36:22.386787 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.386792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.387821 (+ 1029us) server_negotiation.cc:407] Connection header received
0923 14:36:22.388840 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.388843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.388895 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.388969 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:22.390383 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.390915 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.392035 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.392285 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.392386 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.394456 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:22.394459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:22.394462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:22.394484 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:22.395893 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.396617 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.396623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.396627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.396792 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.397346 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.397349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.397351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.397541 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:22.397644 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.398134 (+ 490us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.398258 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:36:22.399173 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.387343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45511 (local address 127.2.218.3:59115)
0923 14:36:22.387702 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.387740 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.388661 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.389140 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.389152 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.389579 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:22.390225 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.390247 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.391043 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.391047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.391926 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.391942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.392421 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.392425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.392567 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.393033 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:22.393048 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:22.394319 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:22.396018 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.396023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.396034 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.396423 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.396980 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.396986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.396989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.397207 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.397649 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:22.397653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:22.397890 (+ 237us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.398072 (+ 182us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.398337 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
I20260923 14:36:22.400192 5160 heartbeater.cc:347] Connected to a master server at 127.2.218.62:45511
I20260923 14:36:22.400386 5160 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:22.400831 5160 heartbeater.cc:511] Master 127.2.218.62:45511 requested a full tablet report, sending...
I20260923 14:36:22.401762 4674 ts_manager.cc:194] Registered new tserver with Master: 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3:34785)
I20260923 14:36:22.402489 4674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:59115
I20260923 14:36:22.403323 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 5029
I20260923 14:36:22.403421 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:22.403628 2920 external_mini_cluster.cc:1428] Setting key 8bf78a3d423a2f50c80cc11dd7376651
I20260923 14:36:22.405545 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:36:22.408934 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.402241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:56159 (local address 127.2.218.62:45511)
0923 14:36:22.402330 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.402333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.402523 (+ 190us) server_negotiation.cc:407] Connection header received
0923 14:36:22.402633 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.402636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.402675 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.402755 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:22.403236 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.403900 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.404846 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.405104 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.405193 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.406183 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:22.406186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:22.406187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:22.406207 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:22.407543 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.407965 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.407967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.407968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.408034 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.408287 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.408289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.408290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.408429 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:22.408500 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.408763 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.408844 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 14:36:22.409056 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.402173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45511 (local address 127.2.218.3:56159)
0923 14:36:22.402446 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.402459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.402554 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.402790 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.402793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.402988 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:22.403100 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.403127 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.404031 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.404035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.404721 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.404749 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.405202 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.405204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.405268 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.405709 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:22.405721 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:22.406016 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:22.407648 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.407653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.407656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.407874 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.408117 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.408120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.408121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.408207 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.408513 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:22.408518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:22.408616 (+ 98us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.408713 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.408841 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":90,"threads_started":1}
Sep 23 14:36:22 dist-test-slave-zl47 krb5kdc[4629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 14:36:22 dist-test-slave-zl47 krb5kdc[4629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:22.428787 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.421834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.2.218.62:45511)
0923 14:36:22.421939 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.421942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.421951 (+ 9us) server_negotiation.cc:407] Connection header received
0923 14:36:22.421976 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.421977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.422003 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.422058 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:22.422692 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.423090 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.423771 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.423988 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.424111 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.426210 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:22.426213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:22.426215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:22.426235 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:22.427410 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.427782 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.427785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.427786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.427858 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.428109 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.428112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.428113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.428251 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:22.428377 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.428568 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.428663 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:36:22.431144 4674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:22.433697 4674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:22.446072 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.440559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:34535 (local address 127.0.0.1:47812)
0923 14:36:22.441001 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.441042 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.441158 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.441893 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.441898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.441919 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:22.442020 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.442026 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.443467 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.443470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.444760 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.444769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.445558 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.445561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.445660 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.445695 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.445809 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.445914 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":50,"thread_start_us":60,"threads_started":1}
I20260923 14:36:22.446281 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.441207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:34785 (local address 127.0.0.1:56418)
0923 14:36:22.441941 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.441953 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.442028 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.442471 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.442473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.442489 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:22.442575 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.442581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.443400 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.443405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.444760 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.444769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.445781 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.445786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.445914 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.445983 (+ 69us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.446043 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.446170 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":155,"thread_start_us":85,"threads_started":1}
I20260923 14:36:22.446337 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.441118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.2.218.2:34535)
0923 14:36:22.441425 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.441435 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.441456 (+ 21us) server_negotiation.cc:407] Connection header received
0923 14:36:22.441525 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.441529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.441711 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.442192 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:22.442312 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.443381 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.444903 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.445450 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.445548 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.445686 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.445919 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.446017 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20260923 14:36:22.446581 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.440835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.2.218.1:46373)
0923 14:36:22.441182 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.441202 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.441793 (+ 591us) server_negotiation.cc:407] Connection header received
0923 14:36:22.441873 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.441877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.442031 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.442192 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:22.442348 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.443368 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.444674 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.445461 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.445557 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.445701 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.446273 (+ 572us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.446441 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":99,"threads_started":1}
I20260923 14:36:22.446702 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.441375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.2.218.3:34785)
0923 14:36:22.441622 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.441637 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.442143 (+ 506us) server_negotiation.cc:407] Connection header received
0923 14:36:22.442201 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.442205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.442355 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.442492 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:22.442663 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.443300 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.444876 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.445578 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.445656 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.445778 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.446475 (+ 697us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.446562 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":103,"threads_started":1}
I20260923 14:36:22.446859 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.440680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:46373 (local address 127.0.0.1:43478)
0923 14:36:22.441686 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.441697 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.441784 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.442171 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.442173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.442188 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:22.442264 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.442268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.443512 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.443515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.444541 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.444551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.445777 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.445780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.445997 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.446016 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.446098 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.446751 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":152,"thread_start_us":82,"threads_started":1}
I20260923 14:36:22.448750 5094 tablet_service.cc:1467] Processing CreateTablet for tablet 0b5e2de10ae94873b001d2a8247b910c (DEFAULT_TABLE table=test-table [id=055e0e1e81704c16a554062a294d8e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:22.448850 4953 tablet_service.cc:1467] Processing CreateTablet for tablet 0b5e2de10ae94873b001d2a8247b910c (DEFAULT_TABLE table=test-table [id=055e0e1e81704c16a554062a294d8e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:22.449486 5094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b5e2de10ae94873b001d2a8247b910c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:22.449352 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 0b5e2de10ae94873b001d2a8247b910c (DEFAULT_TABLE table=test-table [id=055e0e1e81704c16a554062a294d8e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:22.449699 4953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b5e2de10ae94873b001d2a8247b910c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:22.450162 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b5e2de10ae94873b001d2a8247b910c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:22.456761 5185 tablet_bootstrap.cc:492] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: Bootstrap starting.
I20260923 14:36:22.456825 5183 tablet_bootstrap.cc:492] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: Bootstrap starting.
I20260923 14:36:22.458369 5184 tablet_bootstrap.cc:492] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: Bootstrap starting.
I20260923 14:36:22.458906 5185 tablet_bootstrap.cc:654] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:22.459296 5183 tablet_bootstrap.cc:654] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:22.459733 5185 log.cc:824] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:22.459919 5184 tablet_bootstrap.cc:654] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:22.460035 5183 log.cc:824] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:22.460629 5184 log.cc:824] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:22.461413 5185 tablet_bootstrap.cc:492] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: No bootstrap required, opened a new log
I20260923 14:36:22.461627 5185 ts_tablet_manager.cc:1397] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:36:22.462335 5184 tablet_bootstrap.cc:492] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: No bootstrap required, opened a new log
I20260923 14:36:22.462527 5184 ts_tablet_manager.cc:1397] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260923 14:36:22.463137 5183 tablet_bootstrap.cc:492] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: No bootstrap required, opened a new log
I20260923 14:36:22.463325 5183 ts_tablet_manager.cc:1397] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260923 14:36:22.465061 5184 raft_consensus.cc:348] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.465263 5184 raft_consensus.cc:374] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:22.465309 5184 raft_consensus.cc:729] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a751bdab1bed4d528853db392edee82a, State: Initialized, Role: FOLLOWER
I20260923 14:36:22.465305 5185 raft_consensus.cc:348] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.465533 5185 raft_consensus.cc:374] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:22.465606 5185 raft_consensus.cc:729] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b7dfcaf4264ecdba46080a6a4fec12, State: Initialized, Role: FOLLOWER
I20260923 14:36:22.465700 5184 consensus_queue.cc:260] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.466084 5185 consensus_queue.cc:260] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.466276 4879 heartbeater.cc:503] Master 127.2.218.62:45511 was elected leader, sending a full tablet report...
I20260923 14:36:22.466706 5020 heartbeater.cc:503] Master 127.2.218.62:45511 was elected leader, sending a full tablet report...
I20260923 14:36:22.466852 5183 raft_consensus.cc:348] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.467073 5185 ts_tablet_manager.cc:1428] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260923 14:36:22.467082 5183 raft_consensus.cc:374] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:22.467128 5183 raft_consensus.cc:729] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613a5aaa219641a683e0ad1d5005a4b0, State: Initialized, Role: FOLLOWER
I20260923 14:36:22.467526 5183 consensus_queue.cc:260] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.468084 5160 heartbeater.cc:503] Master 127.2.218.62:45511 was elected leader, sending a full tablet report...
I20260923 14:36:22.469352 5183 ts_tablet_manager.cc:1428] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260923 14:36:22.469969 5184 ts_tablet_manager.cc:1428] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20260923 14:36:22.487748 5189 raft_consensus.cc:482] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:22.487885 5189 raft_consensus.cc:504] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.488960 5189 leader_election.cc:288] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535), 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3:34785)
I20260923 14:36:22.493526 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.489219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:56997 (local address 127.2.218.2:34535)
0923 14:36:22.489337 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.489341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.489525 (+ 184us) server_negotiation.cc:407] Connection header received
0923 14:36:22.489651 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.489653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.489700 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.489782 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:22.490140 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.490940 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.492409 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.492953 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.493042 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.493103 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.493358 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.493419 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:36:22.493526 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.489273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:42239 (local address 127.2.218.3:34785)
0923 14:36:22.489362 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.489366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.489529 (+ 163us) server_negotiation.cc:407] Connection header received
0923 14:36:22.489650 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.489653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.489700 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.489782 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:22.490137 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.490940 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.492409 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.492940 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.493046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.493103 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.493358 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.493419 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 14:36:22.493561 4771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.489210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:34785 (local address 127.2.218.1:42239)
0923 14:36:22.489434 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.489448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.489537 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.489800 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.489803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.489836 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:22.490012 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.490018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.491048 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.491051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.492291 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.492301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.493054 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.493057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.493211 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.493264 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.493330 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.493439 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":148,"spinlock_wait_cycles":24448,"thread_start_us":66,"threads_started":1}
I20260923 14:36:22.493566 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.489113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:34535 (local address 127.2.218.1:56997)
0923 14:36:22.489434 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.489448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.489561 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.489800 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.489803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.489819 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:22.490013 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.490018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.491048 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.491051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.492291 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.492301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.493054 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.493057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.493174 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.493261 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.493330 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.493435 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":250,"spinlock_wait_cycles":13056}
I20260923 14:36:22.494063 5114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5e2de10ae94873b001d2a8247b910c" candidate_uuid: "a751bdab1bed4d528853db392edee82a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613a5aaa219641a683e0ad1d5005a4b0" is_pre_election: true
I20260923 14:36:22.494244 4955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5e2de10ae94873b001d2a8247b910c" candidate_uuid: "a751bdab1bed4d528853db392edee82a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" is_pre_election: true
I20260923 14:36:22.494361 5114 raft_consensus.cc:2454] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a751bdab1bed4d528853db392edee82a in term 0.
I20260923 14:36:22.494468 4955 raft_consensus.cc:2454] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a751bdab1bed4d528853db392edee82a in term 0.
I20260923 14:36:22.494777 4764 leader_election.cc:302] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613a5aaa219641a683e0ad1d5005a4b0, a751bdab1bed4d528853db392edee82a; no voters:
I20260923 14:36:22.495008 5189 raft_consensus.cc:2781] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:22.495103 5189 raft_consensus.cc:482] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:22.495162 5189 raft_consensus.cc:3037] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:22.496387 5189 raft_consensus.cc:504] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.496701 5189 leader_election.cc:288] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [CANDIDATE]: Term 1 election: Requested vote from peers 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535), 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3:34785)
I20260923 14:36:22.497006 5114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5e2de10ae94873b001d2a8247b910c" candidate_uuid: "a751bdab1bed4d528853db392edee82a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613a5aaa219641a683e0ad1d5005a4b0"
I20260923 14:36:22.497006 4955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b5e2de10ae94873b001d2a8247b910c" candidate_uuid: "a751bdab1bed4d528853db392edee82a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b7dfcaf4264ecdba46080a6a4fec12"
I20260923 14:36:22.497125 4955 raft_consensus.cc:3037] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:22.497118 5114 raft_consensus.cc:3037] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:22.498377 5114 raft_consensus.cc:2454] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a751bdab1bed4d528853db392edee82a in term 1.
I20260923 14:36:22.498668 4764 leader_election.cc:302] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613a5aaa219641a683e0ad1d5005a4b0, a751bdab1bed4d528853db392edee82a; no voters:
I20260923 14:36:22.498750 4955 raft_consensus.cc:2454] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a751bdab1bed4d528853db392edee82a in term 1.
I20260923 14:36:22.498940 5189 raft_consensus.cc:2781] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:22.499158 5189 raft_consensus.cc:686] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 1 LEADER]: Becoming Leader. State: Replica: a751bdab1bed4d528853db392edee82a, State: Running, Role: LEADER
I20260923 14:36:22.499444 5189 consensus_queue.cc:237] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.502429 4674 catalog_manager.cc:5495] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a reported cstate change: term changed from 0 to 1, leader changed from <none> to a751bdab1bed4d528853db392edee82a (127.2.218.1). New cstate: current_term: 1 leader_uuid: "a751bdab1bed4d528853db392edee82a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
W20260923 14:36:22.529168 4880 tablet.cc:2369] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 14:36:22 dist-test-slave-zl47 krb5kdc[4629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174181, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:22.550547 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.543451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45511 (local address 127.0.0.1:33588)
0923 14:36:22.543716 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.543726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.543860 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.544059 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.544061 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.544384 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:22.544456 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.544462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.545068 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.545072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.545547 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.545554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.545935 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.545938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.546009 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.546606 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:22.546616 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:22.547846 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:22.549147 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.549157 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.549162 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.549367 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.549627 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:22.549631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:22.549633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:22.549704 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:22.550042 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:22.550049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:22.550163 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.550262 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.550395 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":120,"threads_started":1}
I20260923 14:36:22.550546 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.543524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.2.218.62:45511)
0923 14:36:22.543647 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.543652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.543801 (+ 149us) server_negotiation.cc:407] Connection header received
0923 14:36:22.543937 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.543940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.543977 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.544045 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:22.544557 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.544976 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.545661 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.545842 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.545935 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.547959 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:22.547962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:22.547964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:22.547987 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:22.549038 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.549454 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.549459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.549460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.549543 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:22.549790 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:22.549792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:22.549794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:22.549938 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:22.550042 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.550311 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.550440 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":59}
I20260923 14:36:22.553504 4674 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:33588:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 14:36:22.554193 4674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:22.558410 4953 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8dd9c3108148f9bf7ffbb37284894b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=428a43e5ccc64b859b624eb957c1b111]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:36:22.558398 4812 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8dd9c3108148f9bf7ffbb37284894b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=428a43e5ccc64b859b624eb957c1b111]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:36:22.558959 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8dd9c3108148f9bf7ffbb37284894b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:22.558959 4953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8dd9c3108148f9bf7ffbb37284894b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:22.558737 5094 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8dd9c3108148f9bf7ffbb37284894b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=428a43e5ccc64b859b624eb957c1b111]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:36:22.559303 5094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8dd9c3108148f9bf7ffbb37284894b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:22.561313 5184 tablet_bootstrap.cc:492] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: Bootstrap starting.
I20260923 14:36:22.561317 5183 tablet_bootstrap.cc:492] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: Bootstrap starting.
I20260923 14:36:22.561422 5185 tablet_bootstrap.cc:492] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: Bootstrap starting.
I20260923 14:36:22.562223 5184 tablet_bootstrap.cc:654] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:22.562275 5183 tablet_bootstrap.cc:654] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:22.562299 5185 tablet_bootstrap.cc:654] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:22.563342 5184 tablet_bootstrap.cc:492] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: No bootstrap required, opened a new log
I20260923 14:36:22.563433 5184 ts_tablet_manager.cc:1397] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:22.563556 5183 tablet_bootstrap.cc:492] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: No bootstrap required, opened a new log
I20260923 14:36:22.563565 5185 tablet_bootstrap.cc:492] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: No bootstrap required, opened a new log
I20260923 14:36:22.563639 5185 ts_tablet_manager.cc:1397] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:22.563639 5183 ts_tablet_manager.cc:1397] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:22.563809 5184 raft_consensus.cc:348] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.563932 5184 raft_consensus.cc:374] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:22.563967 5184 raft_consensus.cc:729] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a751bdab1bed4d528853db392edee82a, State: Initialized, Role: FOLLOWER
I20260923 14:36:22.564064 5185 raft_consensus.cc:348] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.564040 5183 raft_consensus.cc:348] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.564088 5184 consensus_queue.cc:260] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.564154 5183 raft_consensus.cc:374] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:22.564154 5185 raft_consensus.cc:374] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:22.564186 5183 raft_consensus.cc:729] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613a5aaa219641a683e0ad1d5005a4b0, State: Initialized, Role: FOLLOWER
I20260923 14:36:22.564186 5185 raft_consensus.cc:729] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b7dfcaf4264ecdba46080a6a4fec12, State: Initialized, Role: FOLLOWER
I20260923 14:36:22.564291 5183 consensus_queue.cc:260] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.564291 5185 consensus_queue.cc:260] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.564324 5184 ts_tablet_manager.cc:1428] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260923 14:36:22.564517 5183 ts_tablet_manager.cc:1428] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:22.564517 5185 ts_tablet_manager.cc:1428] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:22.564508 5189 tablet_replica.cc:439] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } } }
I20260923 14:36:22.564611 5189 tablet_replica.cc:442] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:22.564754 5191 tablet_replica.cc:439] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } } }
I20260923 14:36:22.564877 5191 tablet_replica.cc:442] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:22.565063 5190 tablet_replica.cc:439] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } } }
I20260923 14:36:22.565199 5190 tablet_replica.cc:442] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:22.616462 5191 raft_consensus.cc:482] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:22.616552 5191 raft_consensus.cc:504] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.617425 5191 leader_election.cc:288] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a751bdab1bed4d528853db392edee82a (127.2.218.1:46373), 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535)
I20260923 14:36:22.620765 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.617675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:42325 (local address 127.2.218.2:34535)
0923 14:36:22.617783 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.617786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.617969 (+ 183us) server_negotiation.cc:407] Connection header received
0923 14:36:22.618136 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.618142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.618176 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.618245 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:22.618490 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.619011 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.619881 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.620301 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.620398 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.620445 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.620618 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.620674 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:36:22.620775 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.617602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:34535 (local address 127.2.218.3:42325)
0923 14:36:22.617899 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.617909 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.618005 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.618275 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.618277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.618290 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:22.618399 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.618404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.619106 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.619111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.619776 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.619784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.620399 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.620402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.620503 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.620526 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.620606 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.620663 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":4,"spinlock_wait_cycles":10112,"thread_start_us":90,"threads_started":1}
I20260923 14:36:22.620863 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.617711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:48819 (local address 127.2.218.1:46373)
0923 14:36:22.617819 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.617825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.617966 (+ 141us) server_negotiation.cc:407] Connection header received
0923 14:36:22.618157 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.618160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.618194 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.618272 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:22.618514 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.618985 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.619931 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.620345 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.620452 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.620508 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.620688 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.620747 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:36:22.621038 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.617572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:46373 (local address 127.2.218.3:48819)
0923 14:36:22.617884 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.617894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.618023 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.618291 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.618293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.618324 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:22.618423 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.618428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.619083 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.619086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.619818 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.619830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.620464 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.620467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.620577 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.620596 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.620666 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.620725 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":50,"spinlock_wait_cycles":30976}
I20260923 14:36:22.621234 4955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8dd9c3108148f9bf7ffbb37284894b" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" is_pre_election: true
I20260923 14:36:22.621347 4955 raft_consensus.cc:2454] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 0.
I20260923 14:36:22.621637 4832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8dd9c3108148f9bf7ffbb37284894b" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a751bdab1bed4d528853db392edee82a" is_pre_election: true
I20260923 14:36:22.621649 5045 leader_election.cc:302] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b7dfcaf4264ecdba46080a6a4fec12, 613a5aaa219641a683e0ad1d5005a4b0; no voters:
I20260923 14:36:22.621801 4832 raft_consensus.cc:2454] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 0.
I20260923 14:36:22.621873 5191 raft_consensus.cc:2781] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:22.621922 5191 raft_consensus.cc:482] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:22.621958 5191 raft_consensus.cc:3037] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:22.622805 5191 raft_consensus.cc:504] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.623134 5191 leader_election.cc:288] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 election: Requested vote from peers a751bdab1bed4d528853db392edee82a (127.2.218.1:46373), 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535)
I20260923 14:36:22.623399 4955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8dd9c3108148f9bf7ffbb37284894b" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b7dfcaf4264ecdba46080a6a4fec12"
I20260923 14:36:22.623477 4955 raft_consensus.cc:3037] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:22.624091 4955 raft_consensus.cc:2454] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 1.
I20260923 14:36:22.624300 4832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8dd9c3108148f9bf7ffbb37284894b" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a751bdab1bed4d528853db392edee82a"
I20260923 14:36:22.624329 5045 leader_election.cc:302] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b7dfcaf4264ecdba46080a6a4fec12, 613a5aaa219641a683e0ad1d5005a4b0; no voters:
I20260923 14:36:22.624392 4832 raft_consensus.cc:3037] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:22.624512 5191 raft_consensus.cc:2781] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:22.624708 5191 raft_consensus.cc:686] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 LEADER]: Becoming Leader. State: Replica: 613a5aaa219641a683e0ad1d5005a4b0, State: Running, Role: LEADER
I20260923 14:36:22.624974 5191 consensus_queue.cc:237] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.625265 4832 raft_consensus.cc:2454] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 1.
I20260923 14:36:22.626782 5206 tablet_replica.cc:439] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: TxnStatusTablet state changed. Reason: New leader 613a5aaa219641a683e0ad1d5005a4b0. Latest consensus state: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } } }
I20260923 14:36:22.626873 5206 tablet_replica.cc:442] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:36:22.627142 5208 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 14:36:22.627331 5208 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 14:36:22.627419 4672 catalog_manager.cc:5495] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3). New cstate: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260923 14:36:22.645205 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.642128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.2.218.3:34785)
0923 14:36:22.642215 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.642218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.642252 (+ 34us) server_negotiation.cc:407] Connection header received
0923 14:36:22.642406 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.642408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.642448 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.642523 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:22.642746 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.643147 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.644134 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.644301 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.644370 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.644645 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:22.644937 (+ 292us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:22.645014 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.645095 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.645131 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 14:36:22.645277 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.641993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:34785 (local address 127.0.0.1:56420)
0923 14:36:22.642202 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.642214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.642300 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.642552 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.642555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.642561 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:22.642654 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.642659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.643241 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.643244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.644034 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.644043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.644393 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.644396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.644481 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.644522 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:22.645019 (+ 497us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:22.645031 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.645089 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.645167 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
W20260923 14:36:22.647441 5161 tablet.cc:2369] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:22.649412 4955 raft_consensus.cc:1261] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Refusing update from remote peer 613a5aaa219641a683e0ad1d5005a4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:22.649497 4832 raft_consensus.cc:1261] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Refusing update from remote peer 613a5aaa219641a683e0ad1d5005a4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:22.649986 5191 consensus_queue.cc:1035] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:22.650163 5206 consensus_queue.cc:1035] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:22.652587 5189 tablet_replica.cc:439] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: TxnStatusTablet state changed. Reason: New leader 613a5aaa219641a683e0ad1d5005a4b0. Latest consensus state: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } } }
I20260923 14:36:22.652675 5189 tablet_replica.cc:442] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:22.653272 5190 tablet_replica.cc:439] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: TxnStatusTablet state changed. Reason: New leader 613a5aaa219641a683e0ad1d5005a4b0. Latest consensus state: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } } }
I20260923 14:36:22.653365 5190 tablet_replica.cc:442] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:22.654907 5191 tablet_replica.cc:439] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } } }
I20260923 14:36:22.654999 5191 tablet_replica.cc:442] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:36:22.655094 5211 mvcc.cc:205] Tried to move back new op lower bound from 7332553452126969856 to 7332553452034297856. Current Snapshot: MvccSnapshot[applied={T|T < 7332553452126969856}]
I20260923 14:36:22.655457 5189 tablet_replica.cc:439] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } } }
I20260923 14:36:22.655539 5189 tablet_replica.cc:442] T 9b8dd9c3108148f9bf7ffbb37284894b P a751bdab1bed4d528853db392edee82a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:22.656576 5206 tablet_replica.cc:439] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } } }
I20260923 14:36:22.656669 5206 tablet_replica.cc:442] T 9b8dd9c3108148f9bf7ffbb37284894b P 613a5aaa219641a683e0ad1d5005a4b0: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:36:22.657346 5190 tablet_replica.cc:439] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } } }
I20260923 14:36:22.657426 5190 tablet_replica.cc:442] T 9b8dd9c3108148f9bf7ffbb37284894b P 14b7dfcaf4264ecdba46080a6a4fec12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:22.658159 5213 mvcc.cc:205] Tried to move back new op lower bound from 7332553452126969856 to 7332553452034297856. Current Snapshot: MvccSnapshot[applied={T|T < 7332553452126969856}]
I20260923 14:36:22.660593 5212 mvcc.cc:205] Tried to move back new op lower bound from 7332553452126969856 to 7332553452034297856. Current Snapshot: MvccSnapshot[applied={T|T < 7332553452126969856}]
I20260923 14:36:22.665941 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.662850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.2.218.1:46373)
0923 14:36:22.662946 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.662949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.662960 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:36:22.663093 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.663095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.663134 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.663237 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:22.663535 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.663964 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.664743 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.664932 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.665029 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.665314 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:22.665603 (+ 289us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:22.665717 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.665780 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.665827 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:36:22.675369 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.672294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:53379 (local address 127.2.218.3:34785)
0923 14:36:22.672422 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.672428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.672592 (+ 164us) server_negotiation.cc:407] Connection header received
0923 14:36:22.672728 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.672730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.672765 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.672824 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:22.673069 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.673451 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.674525 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.674695 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.674768 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.675054 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:22.675108 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:22.675164 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.675262 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.675295 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:36:22.675438 5223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.672206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:34785 (local address 127.2.218.1:53379)
0923 14:36:22.672503 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.672516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.672635 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.672853 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.672855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.672861 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:22.672963 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.672968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.673552 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.673555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.674417 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.674427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.674793 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.674796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.674876 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.674915 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:22.675188 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:22.675194 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.675250 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.675310 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":108,"threads_started":1}
I20260923 14:36:22.680992 4955 raft_consensus.cc:1261] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Refusing update from remote peer a751bdab1bed4d528853db392edee82a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:22.681200 5114 raft_consensus.cc:1261] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Refusing update from remote peer a751bdab1bed4d528853db392edee82a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:22.681370 5189 consensus_queue.cc:1035] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:22.681887 5189 consensus_queue.cc:1035] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [LEADER]: Connected to new peer: Peer: permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:22.703261 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.700195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:46435 (local address 127.2.218.1:46373)
0923 14:36:22.700311 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.700315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.700329 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:36:22.700372 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.700374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.700409 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.700472 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:22.700686 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.701302 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.702407 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.702599 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.702684 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.702947 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:22.703000 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:22.703047 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.703150 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.703184 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 14:36:22.703322 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.699975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:46373 (local address 127.2.218.3:46435)
0923 14:36:22.700148 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.700161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.700237 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.700491 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.700494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.700500 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:22.700606 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.700612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.701399 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.701402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.702287 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.702297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.702709 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.702712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.702797 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.702840 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:22.703075 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:22.703080 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.703145 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.703211 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 14:36:22.729604 4674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:22.729805 4674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:22.732198 4674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab: Sending DeleteTablet for 3 replicas of tablet 0b5e2de10ae94873b001d2a8247b910c
I20260923 14:36:22.732724 4953 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b5e2de10ae94873b001d2a8247b910c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:22 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:47812
I20260923 14:36:22.732971 5094 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b5e2de10ae94873b001d2a8247b910c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:22 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:56418
I20260923 14:36:22.733222 5231 tablet_replica.cc:330] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: stopping tablet replica
I20260923 14:36:22.733422 4812 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b5e2de10ae94873b001d2a8247b910c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:22 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:43478
I20260923 14:36:22.733496 5231 raft_consensus.cc:2229] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:22.733753 5231 raft_consensus.cc:2258] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:22.734344 4674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:22.734709 4674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:22.735203 5231 ts_tablet_manager.cc:1905] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:22.737833 5231 ts_tablet_manager.cc:1918] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:22.737967 5231 log.cc:1196] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/0b5e2de10ae94873b001d2a8247b910c
I20260923 14:36:22.738307 5231 ts_tablet_manager.cc:1939] T 0b5e2de10ae94873b001d2a8247b910c P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting consensus metadata
I20260923 14:36:22.738799 4953 tablet_service.cc:1467] Processing CreateTablet for tablet fb777f919d514155a69f52499b1ebf4f (DEFAULT_TABLE table=test-table [id=d47cabb52f2349a4b2b8119e3a6db8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:22.738991 4953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb777f919d514155a69f52499b1ebf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:22.739094 5094 tablet_service.cc:1467] Processing CreateTablet for tablet fb777f919d514155a69f52499b1ebf4f (DEFAULT_TABLE table=test-table [id=d47cabb52f2349a4b2b8119e3a6db8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:22.739310 5094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb777f919d514155a69f52499b1ebf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:22.740867 5185 tablet_bootstrap.cc:492] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12: Bootstrap starting.
I20260923 14:36:22.741514 5183 tablet_bootstrap.cc:492] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0: Bootstrap starting.
I20260923 14:36:22.741643 5185 tablet_bootstrap.cc:654] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:22.742700 5183 tablet_bootstrap.cc:654] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:22.743192 4658 catalog_manager.cc:4841] TS 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535): tablet 0b5e2de10ae94873b001d2a8247b910c (table test-table [id=055e0e1e81704c16a554062a294d8e12]) successfully deleted
I20260923 14:36:22.743515 4812 tablet_service.cc:1467] Processing CreateTablet for tablet fb777f919d514155a69f52499b1ebf4f (DEFAULT_TABLE table=test-table [id=d47cabb52f2349a4b2b8119e3a6db8f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:22.743746 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb777f919d514155a69f52499b1ebf4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:22.745903 5184 tablet_bootstrap.cc:492] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a: Bootstrap starting.
I20260923 14:36:22.746718 5184 tablet_bootstrap.cc:654] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:22.747491 5183 tablet_bootstrap.cc:492] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0: No bootstrap required, opened a new log
I20260923 14:36:22.747576 5183 ts_tablet_manager.cc:1397] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260923 14:36:22.747685 5184 tablet_bootstrap.cc:492] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a: No bootstrap required, opened a new log
I20260923 14:36:22.747745 5184 ts_tablet_manager.cc:1397] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:22.747947 5183 raft_consensus.cc:348] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.748039 5183 raft_consensus.cc:374] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:22.748068 5183 raft_consensus.cc:729] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613a5aaa219641a683e0ad1d5005a4b0, State: Initialized, Role: FOLLOWER
I20260923 14:36:22.748087 5184 raft_consensus.cc:348] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.748170 5184 raft_consensus.cc:374] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:22.748194 5184 raft_consensus.cc:729] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a751bdab1bed4d528853db392edee82a, State: Initialized, Role: FOLLOWER
I20260923 14:36:22.748169 5183 consensus_queue.cc:260] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.748286 5184 consensus_queue.cc:260] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.748399 5183 ts_tablet_manager.cc:1428] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:22.748489 5184 ts_tablet_manager.cc:1428] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:22.750547 5233 tablet_replica.cc:330] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: stopping tablet replica
I20260923 14:36:22.750720 5232 tablet_replica.cc:330] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: stopping tablet replica
I20260923 14:36:22.750962 5232 raft_consensus.cc:2229] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:22.751092 5232 raft_consensus.cc:2258] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:22.751377 5233 raft_consensus.cc:2229] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:22.751634 5233 raft_consensus.cc:2258] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:22.752372 5185 tablet_bootstrap.cc:492] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12: No bootstrap required, opened a new log
I20260923 14:36:22.752440 5185 ts_tablet_manager.cc:1397] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260923 14:36:22.752679 5233 ts_tablet_manager.cc:1905] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:22.752794 5185 raft_consensus.cc:348] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.752880 5185 raft_consensus.cc:374] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:22.752911 5185 raft_consensus.cc:729] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b7dfcaf4264ecdba46080a6a4fec12, State: Initialized, Role: FOLLOWER
I20260923 14:36:22.753006 5185 consensus_queue.cc:260] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.753245 5185 ts_tablet_manager.cc:1428] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:22.753425 5232 ts_tablet_manager.cc:1905] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:22.755034 5233 ts_tablet_manager.cc:1918] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:22.755120 5233 log.cc:1196] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/0b5e2de10ae94873b001d2a8247b910c
I20260923 14:36:22.755360 5233 ts_tablet_manager.cc:1939] T 0b5e2de10ae94873b001d2a8247b910c P a751bdab1bed4d528853db392edee82a: Deleting consensus metadata
I20260923 14:36:22.756214 4661 catalog_manager.cc:4841] TS a751bdab1bed4d528853db392edee82a (127.2.218.1:46373): tablet 0b5e2de10ae94873b001d2a8247b910c (table test-table [id=055e0e1e81704c16a554062a294d8e12]) successfully deleted
I20260923 14:36:22.757907 5232 ts_tablet_manager.cc:1918] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:22.758023 5232 log.cc:1196] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/0b5e2de10ae94873b001d2a8247b910c
I20260923 14:36:22.759009 5232 ts_tablet_manager.cc:1939] T 0b5e2de10ae94873b001d2a8247b910c P 613a5aaa219641a683e0ad1d5005a4b0: Deleting consensus metadata
I20260923 14:36:22.759888 4659 catalog_manager.cc:4841] TS 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3:34785): tablet 0b5e2de10ae94873b001d2a8247b910c (table test-table [id=055e0e1e81704c16a554062a294d8e12]) successfully deleted
I20260923 14:36:22.781219 5217 raft_consensus.cc:482] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:22.781301 5217 raft_consensus.cc:504] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.781607 5217 leader_election.cc:288] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a751bdab1bed4d528853db392edee82a (127.2.218.1:46373), 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535)
I20260923 14:36:22.781869 4832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb777f919d514155a69f52499b1ebf4f" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a751bdab1bed4d528853db392edee82a" is_pre_election: true
I20260923 14:36:22.781970 4832 raft_consensus.cc:2454] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 0.
I20260923 14:36:22.782270 5048 leader_election.cc:302] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613a5aaa219641a683e0ad1d5005a4b0, a751bdab1bed4d528853db392edee82a; no voters:
I20260923 14:36:22.782258 4955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb777f919d514155a69f52499b1ebf4f" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" is_pre_election: true
I20260923 14:36:22.782358 4955 raft_consensus.cc:2454] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 0.
I20260923 14:36:22.782403 5217 raft_consensus.cc:2781] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:22.782437 5217 raft_consensus.cc:482] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:22.782466 5217 raft_consensus.cc:3037] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:22.783263 5217 raft_consensus.cc:504] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.783528 5217 leader_election.cc:288] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 election: Requested vote from peers a751bdab1bed4d528853db392edee82a (127.2.218.1:46373), 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535)
I20260923 14:36:22.783818 4832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb777f919d514155a69f52499b1ebf4f" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a751bdab1bed4d528853db392edee82a"
I20260923 14:36:22.783823 4955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb777f919d514155a69f52499b1ebf4f" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b7dfcaf4264ecdba46080a6a4fec12"
I20260923 14:36:22.783906 4832 raft_consensus.cc:3037] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:22.783906 4955 raft_consensus.cc:3037] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:22.784727 4832 raft_consensus.cc:2454] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 1.
I20260923 14:36:22.784809 4955 raft_consensus.cc:2454] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 1.
I20260923 14:36:22.785037 5048 leader_election.cc:302] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613a5aaa219641a683e0ad1d5005a4b0, a751bdab1bed4d528853db392edee82a; no voters:
I20260923 14:36:22.785229 5217 raft_consensus.cc:2781] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:22.785308 5217 raft_consensus.cc:686] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 LEADER]: Becoming Leader. State: Replica: 613a5aaa219641a683e0ad1d5005a4b0, State: Running, Role: LEADER
I20260923 14:36:22.785409 5217 consensus_queue.cc:237] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:22.786728 4673 catalog_manager.cc:5495] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3). New cstate: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260923 14:36:22.808789 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.805964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.2.218.3:34785)
0923 14:36:22.806064 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.806070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.806089 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:36:22.806224 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.806226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.806262 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.806323 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:22.806537 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.806953 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.807943 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.808133 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.808209 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.808436 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:22.808493 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:22.808544 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.808633 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.808669 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:36:22.816268 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.813663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:56531 (local address 127.2.218.3:34785)
0923 14:36:22.813752 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:22.813754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:22.813766 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:22.813831 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:22.813833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:22.813863 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:22.813920 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:22.814139 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.814529 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.815473 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.815642 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.815715 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.815936 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:22.815983 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:22.816033 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:22.816150 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 14:36:22.816180 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 14:36:22.816344 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:22.813585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:34785 (local address 127.2.218.3:56531)
0923 14:36:22.813694 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:22.813705 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:22.813766 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:22.813944 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:22.813947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:22.813953 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:22.814025 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.814029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.814627 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.814630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.815374 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:22.815383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:22.815742 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:22.815745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:22.815815 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:22.815840 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:22.816054 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:22.816059 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:36:22.816144 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 14:36:22.816206 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 14:36:22.820785 4832 raft_consensus.cc:1261] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Refusing update from remote peer 613a5aaa219641a683e0ad1d5005a4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:22.820825 4955 raft_consensus.cc:1261] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Refusing update from remote peer 613a5aaa219641a683e0ad1d5005a4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:22.821241 5217 consensus_queue.cc:1035] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:22.821399 5206 consensus_queue.cc:1035] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 14:36:22.836453 5227 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 14:36:24.388403 4674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:24.388566 4674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:24.390738 4674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab: Sending DeleteTablet for 3 replicas of tablet fb777f919d514155a69f52499b1ebf4f
I20260923 14:36:24.391227 4812 tablet_service.cc:1514] Processing DeleteTablet for tablet fb777f919d514155a69f52499b1ebf4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:24 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:43478
I20260923 14:36:24.391342 5094 tablet_service.cc:1514] Processing DeleteTablet for tablet fb777f919d514155a69f52499b1ebf4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:24 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:56418
I20260923 14:36:24.391522 5262 tablet_replica.cc:330] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a: stopping tablet replica
I20260923 14:36:24.391512 4953 tablet_service.cc:1514] Processing DeleteTablet for tablet fb777f919d514155a69f52499b1ebf4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:24 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:47812
I20260923 14:36:24.391669 5262 raft_consensus.cc:2229] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:24.391794 5262 raft_consensus.cc:2258] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:24.394172 5262 ts_tablet_manager.cc:1905] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:24.394280 5264 tablet_replica.cc:330] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12: stopping tablet replica
I20260923 14:36:24.394397 5264 raft_consensus.cc:2229] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:24.394501 5264 raft_consensus.cc:2258] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:24.394698 5263 tablet_replica.cc:330] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0: stopping tablet replica
I20260923 14:36:24.394809 5263 raft_consensus.cc:2229] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:24.395072 5263 raft_consensus.cc:2258] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:24.395562 5264 ts_tablet_manager.cc:1905] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:24.396737 5262 ts_tablet_manager.cc:1918] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 14:36:24.396823 5262 log.cc:1196] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/fb777f919d514155a69f52499b1ebf4f
I20260923 14:36:24.397104 5262 ts_tablet_manager.cc:1939] T fb777f919d514155a69f52499b1ebf4f P a751bdab1bed4d528853db392edee82a: Deleting consensus metadata
I20260923 14:36:24.398159 5264 ts_tablet_manager.cc:1918] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 14:36:24.398238 5264 log.cc:1196] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/fb777f919d514155a69f52499b1ebf4f
I20260923 14:36:24.398550 5264 ts_tablet_manager.cc:1939] T fb777f919d514155a69f52499b1ebf4f P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting consensus metadata
I20260923 14:36:24.398699 4661 catalog_manager.cc:4841] TS a751bdab1bed4d528853db392edee82a (127.2.218.1:46373): tablet fb777f919d514155a69f52499b1ebf4f (table test-table [id=d47cabb52f2349a4b2b8119e3a6db8f6]) successfully deleted
I20260923 14:36:24.399266 4658 catalog_manager.cc:4841] TS 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535): tablet fb777f919d514155a69f52499b1ebf4f (table test-table [id=d47cabb52f2349a4b2b8119e3a6db8f6]) successfully deleted
I20260923 14:36:24.399529 5263 ts_tablet_manager.cc:1905] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:24.401809 5263 ts_tablet_manager.cc:1918] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 14:36:24.401906 5263 log.cc:1196] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/fb777f919d514155a69f52499b1ebf4f
I20260923 14:36:24.402240 5263 ts_tablet_manager.cc:1939] T fb777f919d514155a69f52499b1ebf4f P 613a5aaa219641a683e0ad1d5005a4b0: Deleting consensus metadata
I20260923 14:36:24.402810 4659 catalog_manager.cc:4841] TS 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3:34785): tablet fb777f919d514155a69f52499b1ebf4f (table test-table [id=d47cabb52f2349a4b2b8119e3a6db8f6]) successfully deleted
Sep 23 14:36:24 dist-test-slave-zl47 krb5kdc[4629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 14:36:24 dist-test-slave-zl47 krb5kdc[4629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174184, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:24.419772 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:24.412103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.2.218.62:45511)
0923 14:36:24.412364 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:24.412370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:24.412383 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:36:24.412514 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:24.412517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:24.412575 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:24.412647 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:24.413222 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:24.413765 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:24.414524 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:24.414731 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:24.414808 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:24.416919 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:24.416922 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:24.416924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:24.416945 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:24.418448 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:24.418768 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:24.418770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:24.418771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:24.418844 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:24.419040 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:24.419042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:24.419042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:24.419238 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:24.419335 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:24.419478 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 14:36:24.419593 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260923 14:36:24.421886 4674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:24.422281 4674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:24.425906 4812 tablet_service.cc:1467] Processing CreateTablet for tablet a4ad006a748e4fb59cfd66a7dda8ddb1 (DEFAULT_TABLE table=test-table [id=e2d4737cd69645aea21c11ed1ec591a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:24.425923 4953 tablet_service.cc:1467] Processing CreateTablet for tablet a4ad006a748e4fb59cfd66a7dda8ddb1 (DEFAULT_TABLE table=test-table [id=e2d4737cd69645aea21c11ed1ec591a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:24.425980 5094 tablet_service.cc:1467] Processing CreateTablet for tablet a4ad006a748e4fb59cfd66a7dda8ddb1 (DEFAULT_TABLE table=test-table [id=e2d4737cd69645aea21c11ed1ec591a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:24.426179 4953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ad006a748e4fb59cfd66a7dda8ddb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:24.426191 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ad006a748e4fb59cfd66a7dda8ddb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:24.426208 5094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ad006a748e4fb59cfd66a7dda8ddb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:24.427861 5273 tablet_bootstrap.cc:492] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a: Bootstrap starting.
I20260923 14:36:24.428224 5274 tablet_bootstrap.cc:492] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0: Bootstrap starting.
I20260923 14:36:24.428588 5275 tablet_bootstrap.cc:492] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: Bootstrap starting.
I20260923 14:36:24.428727 5273 tablet_bootstrap.cc:654] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:24.428927 5274 tablet_bootstrap.cc:654] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:24.429373 5275 tablet_bootstrap.cc:654] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:24.429785 5273 tablet_bootstrap.cc:492] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a: No bootstrap required, opened a new log
I20260923 14:36:24.429872 5273 ts_tablet_manager.cc:1397] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:24.429965 5274 tablet_bootstrap.cc:492] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0: No bootstrap required, opened a new log
I20260923 14:36:24.430047 5274 ts_tablet_manager.cc:1397] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:24.430300 5273 raft_consensus.cc:348] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.430411 5273 raft_consensus.cc:374] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:24.430441 5273 raft_consensus.cc:729] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a751bdab1bed4d528853db392edee82a, State: Initialized, Role: FOLLOWER
I20260923 14:36:24.430524 5274 raft_consensus.cc:348] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.430575 5273 consensus_queue.cc:260] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.430642 5274 raft_consensus.cc:374] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:24.430681 5274 raft_consensus.cc:729] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613a5aaa219641a683e0ad1d5005a4b0, State: Initialized, Role: FOLLOWER
I20260923 14:36:24.430729 5275 tablet_bootstrap.cc:492] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: No bootstrap required, opened a new log
I20260923 14:36:24.430789 5275 ts_tablet_manager.cc:1397] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:24.430787 5274 consensus_queue.cc:260] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.430902 5273 ts_tablet_manager.cc:1428] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:24.431048 5274 ts_tablet_manager.cc:1428] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 14:36:24.431142 5275 raft_consensus.cc:348] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.431222 5275 raft_consensus.cc:374] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:24.431248 5275 raft_consensus.cc:729] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b7dfcaf4264ecdba46080a6a4fec12, State: Initialized, Role: FOLLOWER
I20260923 14:36:24.431348 5275 consensus_queue.cc:260] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.431663 5275 ts_tablet_manager.cc:1428] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:24.444106 5241 raft_consensus.cc:482] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:24.444182 5241 raft_consensus.cc:504] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.444545 5241 leader_election.cc:288] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a751bdab1bed4d528853db392edee82a (127.2.218.1:46373), 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535)
I20260923 14:36:24.444790 4830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ad006a748e4fb59cfd66a7dda8ddb1" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a751bdab1bed4d528853db392edee82a" is_pre_election: true
I20260923 14:36:24.444828 4955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ad006a748e4fb59cfd66a7dda8ddb1" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" is_pre_election: true
I20260923 14:36:24.444907 4830 raft_consensus.cc:2454] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 0.
I20260923 14:36:24.444960 4955 raft_consensus.cc:2454] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 0.
I20260923 14:36:24.445233 5048 leader_election.cc:302] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613a5aaa219641a683e0ad1d5005a4b0, a751bdab1bed4d528853db392edee82a; no voters:
I20260923 14:36:24.445372 5241 raft_consensus.cc:2781] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:24.445421 5241 raft_consensus.cc:482] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:24.445466 5241 raft_consensus.cc:3037] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:24.446233 5241 raft_consensus.cc:504] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.446522 5241 leader_election.cc:288] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 election: Requested vote from peers a751bdab1bed4d528853db392edee82a (127.2.218.1:46373), 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535)
I20260923 14:36:24.446784 4830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ad006a748e4fb59cfd66a7dda8ddb1" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a751bdab1bed4d528853db392edee82a"
I20260923 14:36:24.446815 4955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ad006a748e4fb59cfd66a7dda8ddb1" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b7dfcaf4264ecdba46080a6a4fec12"
I20260923 14:36:24.446892 4955 raft_consensus.cc:3037] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:24.446902 4830 raft_consensus.cc:3037] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:24.447515 4955 raft_consensus.cc:2454] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 1.
I20260923 14:36:24.447686 4830 raft_consensus.cc:2454] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 1.
I20260923 14:36:24.447726 5045 leader_election.cc:302] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b7dfcaf4264ecdba46080a6a4fec12, 613a5aaa219641a683e0ad1d5005a4b0; no voters:
I20260923 14:36:24.447858 5241 raft_consensus.cc:2781] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:24.447932 5241 raft_consensus.cc:686] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 LEADER]: Becoming Leader. State: Replica: 613a5aaa219641a683e0ad1d5005a4b0, State: Running, Role: LEADER
I20260923 14:36:24.448032 5241 consensus_queue.cc:237] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.449169 4672 catalog_manager.cc:5495] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3). New cstate: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260923 14:36:24.460466 5283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:24.458053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.2.218.3:34785)
0923 14:36:24.458258 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:24.458270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:24.458280 (+ 10us) server_negotiation.cc:407] Connection header received
0923 14:36:24.458309 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:24.458310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:24.458341 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:24.458400 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:24.458600 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:24.459010 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:24.459733 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:24.459911 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:24.459975 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:24.460138 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:24.460187 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:24.460227 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:24.460295 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 14:36:24.460328 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":106,"threads_started":1}
W20260923 14:36:24.464453 5021 tablet.cc:2369] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:24.464802 4955 raft_consensus.cc:1261] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Refusing update from remote peer 613a5aaa219641a683e0ad1d5005a4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:24.464895 4830 raft_consensus.cc:1261] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Refusing update from remote peer 613a5aaa219641a683e0ad1d5005a4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:24.465140 5256 consensus_queue.cc:1035] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:24.465253 5241 consensus_queue.cc:1035] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:24.487673 4674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:24.487768 4674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:24.489260 4674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab: Sending DeleteTablet for 3 replicas of tablet a4ad006a748e4fb59cfd66a7dda8ddb1
I20260923 14:36:24.489657 4812 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ad006a748e4fb59cfd66a7dda8ddb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:24 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:43478
I20260923 14:36:24.489699 4953 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ad006a748e4fb59cfd66a7dda8ddb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:24 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:47812
I20260923 14:36:24.489804 5262 tablet_replica.cc:330] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a: stopping tablet replica
I20260923 14:36:24.489840 5264 tablet_replica.cc:330] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: stopping tablet replica
I20260923 14:36:24.489823 5093 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ad006a748e4fb59cfd66a7dda8ddb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:24 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:56418
I20260923 14:36:24.489898 5262 raft_consensus.cc:2229] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:24.489921 5264 raft_consensus.cc:2229] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:24.489984 5262 raft_consensus.cc:2258] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:24.489988 5264 raft_consensus.cc:2258] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:24.490159 5263 tablet_replica.cc:330] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0: stopping tablet replica
I20260923 14:36:24.490253 5263 raft_consensus.cc:2229] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:24.490386 5263 raft_consensus.cc:2258] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:24.490845 5262 ts_tablet_manager.cc:1905] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:24.490913 5263 ts_tablet_manager.cc:1905] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:24.490913 5264 ts_tablet_manager.cc:1905] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:24.492588 5262 ts_tablet_manager.cc:1918] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:24.492667 5262 log.cc:1196] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/a4ad006a748e4fb59cfd66a7dda8ddb1
I20260923 14:36:24.492861 5264 ts_tablet_manager.cc:1918] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:24.492861 5263 ts_tablet_manager.cc:1918] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:24.492894 5262 ts_tablet_manager.cc:1939] T a4ad006a748e4fb59cfd66a7dda8ddb1 P a751bdab1bed4d528853db392edee82a: Deleting consensus metadata
I20260923 14:36:24.492928 5264 log.cc:1196] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/a4ad006a748e4fb59cfd66a7dda8ddb1
I20260923 14:36:24.492928 5263 log.cc:1196] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/a4ad006a748e4fb59cfd66a7dda8ddb1
I20260923 14:36:24.493150 5263 ts_tablet_manager.cc:1939] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 613a5aaa219641a683e0ad1d5005a4b0: Deleting consensus metadata
I20260923 14:36:24.493155 5264 ts_tablet_manager.cc:1939] T a4ad006a748e4fb59cfd66a7dda8ddb1 P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting consensus metadata
I20260923 14:36:24.493438 4661 catalog_manager.cc:4841] TS a751bdab1bed4d528853db392edee82a (127.2.218.1:46373): tablet a4ad006a748e4fb59cfd66a7dda8ddb1 (table test-table [id=e2d4737cd69645aea21c11ed1ec591a5]) successfully deleted
I20260923 14:36:24.493656 4659 catalog_manager.cc:4841] TS 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3:34785): tablet a4ad006a748e4fb59cfd66a7dda8ddb1 (table test-table [id=e2d4737cd69645aea21c11ed1ec591a5]) successfully deleted
I20260923 14:36:24.493642 4674 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:24.493971 4674 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:24.494240 4658 catalog_manager.cc:4841] TS 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535): tablet a4ad006a748e4fb59cfd66a7dda8ddb1 (table test-table [id=e2d4737cd69645aea21c11ed1ec591a5]) successfully deleted
I20260923 14:36:24.497614 4812 tablet_service.cc:1467] Processing CreateTablet for tablet c7881bf3d5884c5da76099b1c9ffa76d (DEFAULT_TABLE table=test-table [id=09b59b812fa34464b6cf19d050c1c422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:24.497644 4953 tablet_service.cc:1467] Processing CreateTablet for tablet c7881bf3d5884c5da76099b1c9ffa76d (DEFAULT_TABLE table=test-table [id=09b59b812fa34464b6cf19d050c1c422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:24.497716 5093 tablet_service.cc:1467] Processing CreateTablet for tablet c7881bf3d5884c5da76099b1c9ffa76d (DEFAULT_TABLE table=test-table [id=09b59b812fa34464b6cf19d050c1c422]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:24.497792 4812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7881bf3d5884c5da76099b1c9ffa76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:24.497859 4953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7881bf3d5884c5da76099b1c9ffa76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:24.497891 5093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7881bf3d5884c5da76099b1c9ffa76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:24.499333 5273 tablet_bootstrap.cc:492] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a: Bootstrap starting.
I20260923 14:36:24.499717 5275 tablet_bootstrap.cc:492] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12: Bootstrap starting.
I20260923 14:36:24.500031 5273 tablet_bootstrap.cc:654] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:24.500558 5275 tablet_bootstrap.cc:654] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:24.501024 5273 tablet_bootstrap.cc:492] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a: No bootstrap required, opened a new log
I20260923 14:36:24.501104 5273 ts_tablet_manager.cc:1397] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:24.501142 5274 tablet_bootstrap.cc:492] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0: Bootstrap starting.
I20260923 14:36:24.501410 5273 raft_consensus.cc:348] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.501530 5273 raft_consensus.cc:374] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:24.501559 5273 raft_consensus.cc:729] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a751bdab1bed4d528853db392edee82a, State: Initialized, Role: FOLLOWER
I20260923 14:36:24.501648 5275 tablet_bootstrap.cc:492] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12: No bootstrap required, opened a new log
I20260923 14:36:24.501713 5275 ts_tablet_manager.cc:1397] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:24.501668 5273 consensus_queue.cc:260] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.501812 5274 tablet_bootstrap.cc:654] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:24.501925 5273 ts_tablet_manager.cc:1428] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:24.502043 5275 raft_consensus.cc:348] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.502156 5275 raft_consensus.cc:374] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:24.502195 5275 raft_consensus.cc:729] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b7dfcaf4264ecdba46080a6a4fec12, State: Initialized, Role: FOLLOWER
I20260923 14:36:24.502285 5275 consensus_queue.cc:260] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.502482 5275 ts_tablet_manager.cc:1428] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:24.503268 5274 tablet_bootstrap.cc:492] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0: No bootstrap required, opened a new log
I20260923 14:36:24.503345 5274 ts_tablet_manager.cc:1397] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:24.503687 5274 raft_consensus.cc:348] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.503798 5274 raft_consensus.cc:374] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:24.503831 5274 raft_consensus.cc:729] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613a5aaa219641a683e0ad1d5005a4b0, State: Initialized, Role: FOLLOWER
I20260923 14:36:24.503937 5274 consensus_queue.cc:260] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.504187 5274 ts_tablet_manager.cc:1428] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 14:36:24.615567 5249 raft_consensus.cc:482] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:24.615684 5249 raft_consensus.cc:504] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.616007 5249 leader_election.cc:288] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a751bdab1bed4d528853db392edee82a (127.2.218.1:46373), 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535)
I20260923 14:36:24.616328 4955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7881bf3d5884c5da76099b1c9ffa76d" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" is_pre_election: true
I20260923 14:36:24.616348 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7881bf3d5884c5da76099b1c9ffa76d" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a751bdab1bed4d528853db392edee82a" is_pre_election: true
I20260923 14:36:24.616429 4955 raft_consensus.cc:2454] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 0.
I20260923 14:36:24.616483 4831 raft_consensus.cc:2454] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 0.
I20260923 14:36:24.616722 5045 leader_election.cc:302] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b7dfcaf4264ecdba46080a6a4fec12, 613a5aaa219641a683e0ad1d5005a4b0; no voters:
I20260923 14:36:24.616914 5249 raft_consensus.cc:2781] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:24.616971 5249 raft_consensus.cc:482] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:24.617025 5249 raft_consensus.cc:3037] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:24.617897 5249 raft_consensus.cc:504] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.618211 5249 leader_election.cc:288] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 election: Requested vote from peers a751bdab1bed4d528853db392edee82a (127.2.218.1:46373), 14b7dfcaf4264ecdba46080a6a4fec12 (127.2.218.2:34535)
I20260923 14:36:24.618527 4831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7881bf3d5884c5da76099b1c9ffa76d" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a751bdab1bed4d528853db392edee82a"
I20260923 14:36:24.618526 4955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7881bf3d5884c5da76099b1c9ffa76d" candidate_uuid: "613a5aaa219641a683e0ad1d5005a4b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b7dfcaf4264ecdba46080a6a4fec12"
I20260923 14:36:24.618633 4831 raft_consensus.cc:3037] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:24.618643 4955 raft_consensus.cc:3037] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:24.619493 4955 raft_consensus.cc:2454] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 1.
I20260923 14:36:24.619554 4831 raft_consensus.cc:2454] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613a5aaa219641a683e0ad1d5005a4b0 in term 1.
I20260923 14:36:24.619787 5048 leader_election.cc:302] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613a5aaa219641a683e0ad1d5005a4b0, a751bdab1bed4d528853db392edee82a; no voters:
I20260923 14:36:24.619987 5249 raft_consensus.cc:2781] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:24.620078 5249 raft_consensus.cc:686] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 LEADER]: Becoming Leader. State: Replica: 613a5aaa219641a683e0ad1d5005a4b0, State: Running, Role: LEADER
I20260923 14:36:24.620187 5249 consensus_queue.cc:237] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } }
I20260923 14:36:24.621357 4673 catalog_manager.cc:5495] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3). New cstate: current_term: 1 leader_uuid: "613a5aaa219641a683e0ad1d5005a4b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613a5aaa219641a683e0ad1d5005a4b0" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260923 14:36:24.659447 4830 raft_consensus.cc:1261] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a [term 1 FOLLOWER]: Refusing update from remote peer 613a5aaa219641a683e0ad1d5005a4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:24.659611 4955 raft_consensus.cc:1261] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Refusing update from remote peer 613a5aaa219641a683e0ad1d5005a4b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:24.659780 5249 consensus_queue.cc:1035] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a751bdab1bed4d528853db392edee82a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:24.660020 5241 consensus_queue.cc:1035] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b7dfcaf4264ecdba46080a6a4fec12" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 14:36:24.671289 5242 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 14:36:26.214953 4674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:26.215123 4674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:26.220057 4674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66855d46195b4de294ff9caaf1ac7cab: Sending DeleteTablet for 3 replicas of tablet c7881bf3d5884c5da76099b1c9ffa76d
I20260923 14:36:26.220566 4953 tablet_service.cc:1514] Processing DeleteTablet for tablet c7881bf3d5884c5da76099b1c9ffa76d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:26 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:47812
I20260923 14:36:26.220604 4812 tablet_service.cc:1514] Processing DeleteTablet for tablet c7881bf3d5884c5da76099b1c9ffa76d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:26 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:43478
I20260923 14:36:26.220669 5093 tablet_service.cc:1514] Processing DeleteTablet for tablet c7881bf3d5884c5da76099b1c9ffa76d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:26 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:56418
I20260923 14:36:26.220999 5313 tablet_replica.cc:330] T c7881bf3d5884c5da76099b1c9ffa76d P a751bdab1bed4d528853db392edee82a: stopping tablet replica
I20260923 14:36:26.221089 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 4747
I20260923 14:36:26.222286 5314 tablet_replica.cc:330] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0: stopping tablet replica
I20260923 14:36:26.222421 5314 raft_consensus.cc:2229] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:26.222579 5314 raft_consensus.cc:2258] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:26.223165 5314 ts_tablet_manager.cc:1905] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:26.224983 5314 ts_tablet_manager.cc:1918] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 14:36:26.225064 5314 log.cc:1196] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/c7881bf3d5884c5da76099b1c9ffa76d
I20260923 14:36:26.225188 5312 tablet_replica.cc:330] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12: stopping tablet replica
I20260923 14:36:26.225312 5312 raft_consensus.cc:2229] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:26.225332 5314 ts_tablet_manager.cc:1939] T c7881bf3d5884c5da76099b1c9ffa76d P 613a5aaa219641a683e0ad1d5005a4b0: Deleting consensus metadata
I20260923 14:36:26.225415 5312 raft_consensus.cc:2258] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:26.225878 4659 catalog_manager.cc:4841] TS 613a5aaa219641a683e0ad1d5005a4b0 (127.2.218.3:34785): tablet c7881bf3d5884c5da76099b1c9ffa76d (table test-table [id=09b59b812fa34464b6cf19d050c1c422]) successfully deleted
I20260923 14:36:26.225977 5312 ts_tablet_manager.cc:1905] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 14:36:26.228777 4661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:46373 (error 108)
I20260923 14:36:26.228861 5312 ts_tablet_manager.cc:1918] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 14:36:26.228928 5312 log.cc:1196] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/c7881bf3d5884c5da76099b1c9ffa76d
I20260923 14:36:26.228942 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 4888
I20260923 14:36:26.229167 5312 ts_tablet_manager.cc:1939] T c7881bf3d5884c5da76099b1c9ffa76d P 14b7dfcaf4264ecdba46080a6a4fec12: Deleting consensus metadata
I20260923 14:36:26.230505 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:26.230154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:46373 (local address 127.0.0.1:44648)
0923 14:36:26.230368 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:26.230434 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:46373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
W20260923 14:36:26.230624 4661 catalog_manager.cc:4584] TS a751bdab1bed4d528853db392edee82a (127.2.218.1:46373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7881bf3d5884c5da76099b1c9ffa76d: Network error: Client connection negotiation failed: client connection to 127.2.218.1:46373: connect: Connection refused (error 111)
I20260923 14:36:26.234361 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 5029
I20260923 14:36:26.235679 5316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:26.234269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:34535 (local address 127.0.0.1:47466)
0923 14:36:26.234362 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:26.234419 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.2:34535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 14:36:26.239830 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 4642
2026-09-23T14:36:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5608 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:26 dist-test-slave-zl47 krb5kdc[5319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:26 dist-test-slave-zl47 krb5kdc[5319](info): set up 2 sockets
Sep 23 14:36:26 dist-test-slave-zl47 krb5kdc[5319](info): commencing operation
krb5kdc: starting...
W20260923 14:36:27.133900 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.873s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:27 dist-test-slave-zl47 krb5kdc[5319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:27.234617 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36197
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:36197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:27.327485 5332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:27.327677 5332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:27.327735 5332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:27.331151 5332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:27.331198 5332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:27.331240 5332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:27.331274 5332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:27.331302 5332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:27.335534 5332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:36197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:27.336269 5332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:27.337035 5332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:27.343876 5338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:27.343880 5337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:27.343851 5340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:27.344100 5332 server_base.cc:1034] running on GCE node
I20260923 14:36:27.344551 5332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:27.345369 5332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:27.346549 5332 hybrid_clock.cc:648] HybridClock initialized: now 1790174187346534 us; error 26 us; skew 500 ppm
Sep 23 14:36:27 dist-test-slave-zl47 krb5kdc[5319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174187, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:27.348707 5332 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:27.349323 5332 webserver.cc:460] Webserver started at http://127.2.218.62:35409/ using document root <none> and password file <none>
I20260923 14:36:27.349699 5332 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:27.349747 5332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:27.349871 5332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:27.351389 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "4b3333bc968a4cdaab74b97f373b31f0"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "3c7f2ad5a3711b44709bbb8995d731f5"
server_key_iv: "dbf3343ab0f17e2ac4f2b9bb467bed77"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.351791 5332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "4b3333bc968a4cdaab74b97f373b31f0"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "3c7f2ad5a3711b44709bbb8995d731f5"
server_key_iv: "dbf3343ab0f17e2ac4f2b9bb467bed77"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.354592 5332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:36:27.356384 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:27.357372 5332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:36:27.357489 5332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "4b3333bc968a4cdaab74b97f373b31f0"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "3c7f2ad5a3711b44709bbb8995d731f5"
server_key_iv: "dbf3343ab0f17e2ac4f2b9bb467bed77"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.357577 5332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:27.390021 5332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:27.392666 5332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:27.392828 5332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:27.398332 5332 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:36197
I20260923 14:36:27.398388 5399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:36197 every 8 connection(s)
I20260923 14:36:27.399063 5332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:27.401243 5400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:27.405985 5400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0: Bootstrap starting.
I20260923 14:36:27.408293 5400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:27.408910 5400 log.cc:824] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:27.409283 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 5332
I20260923 14:36:27.409408 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:27.409663 2920 external_mini_cluster.cc:1428] Setting key 165500ff895b316e5ab191a3bffd1bdf
I20260923 14:36:27.410928 5400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0: No bootstrap required, opened a new log
I20260923 14:36:27.415114 5400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3333bc968a4cdaab74b97f373b31f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36197 } }
I20260923 14:36:27.415359 5400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:27.415444 5400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3333bc968a4cdaab74b97f373b31f0, State: Initialized, Role: FOLLOWER
I20260923 14:36:27.415889 5400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3333bc968a4cdaab74b97f373b31f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36197 } }
I20260923 14:36:27.416033 5400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:27.416133 5400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:27.416230 5400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:27.417441 5400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3333bc968a4cdaab74b97f373b31f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36197 } }
I20260923 14:36:27.417817 5400 leader_election.cc:302] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b3333bc968a4cdaab74b97f373b31f0; no voters:
I20260923 14:36:27.418213 5400 leader_election.cc:288] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:27.418288 5405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:27.418568 5405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [term 1 LEADER]: Becoming Leader. State: Replica: 4b3333bc968a4cdaab74b97f373b31f0, State: Running, Role: LEADER
Sep 23 14:36:27 dist-test-slave-zl47 krb5kdc[5319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:27.418865 5405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3333bc968a4cdaab74b97f373b31f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36197 } }
I20260923 14:36:27.419461 5400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:27.420224 5407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b3333bc968a4cdaab74b97f373b31f0. Latest consensus state: current_term: 1 leader_uuid: "4b3333bc968a4cdaab74b97f373b31f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3333bc968a4cdaab74b97f373b31f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36197 } } }
I20260923 14:36:27.420351 5407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:27.420958 5406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b3333bc968a4cdaab74b97f373b31f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b3333bc968a4cdaab74b97f373b31f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36197 } } }
I20260923 14:36:27.421047 5406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:27.421070 5413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:27.423761 5413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:27.424717 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:27.410646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.2.218.62:36197)
0923 14:36:27.410953 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:27.410980 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:27.411016 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:36:27.411520 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:27.411533 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:27.411782 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:27.412005 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:27.412913 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:27.414368 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:27.415549 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:27.416144 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:27.416299 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:27.419501 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:27.419508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:27.419522 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:27.419555 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:27.421640 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:27.422111 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:27.422119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:27.422125 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:27.422269 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:27.422598 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:27.422601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:27.422603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:27.423031 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:27.423213 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:27.423724 (+ 511us) server_negotiation.cc:299] Negotiation successful
0923 14:36:27.423968 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260923 14:36:27.430227 5413 catalog_manager.cc:1347] Generated new cluster ID: f11dc98b42b0433598fc5b593f5a00c5
I20260923 14:36:27.430294 5413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:27.448923 5413 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:27.449968 5413 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:27.456558 5413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b3333bc968a4cdaab74b97f373b31f0: Generated new TSK 0
I20260923 14:36:27.457176 5413 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:36:27.484931 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36197
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:27.578892 5426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:27.579066 5426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:27.579105 5426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:27.582532 5426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:27.582571 5426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:27.582623 5426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:27.586707 5426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:27.587373 5426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:27.588063 5426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:27.595850 5432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:27.595899 5426 server_base.cc:1034] running on GCE node
W20260923 14:36:27.595880 5434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:27.596045 5431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:27.596436 5426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:27.596902 5426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:27.598049 5426 hybrid_clock.cc:648] HybridClock initialized: now 1790174187598040 us; error 26 us; skew 500 ppm
Sep 23 14:36:27 dist-test-slave-zl47 krb5kdc[5319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174187, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:27.600419 5426 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:36:27.601095 5426 webserver.cc:460] Webserver started at http://127.2.218.1:45053/ using document root <none> and password file <none>
I20260923 14:36:27.601482 5426 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:27.601527 5426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:27.601652 5426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:27.603281 5426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "c113ebc2317546e6a7ad55883caaef53"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "3ef2aee71e218a458240663253bbeffe"
server_key_iv: "3fcb7763097e9e8294f78a6438cde17b"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.603678 5426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "c113ebc2317546e6a7ad55883caaef53"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "3ef2aee71e218a458240663253bbeffe"
server_key_iv: "3fcb7763097e9e8294f78a6438cde17b"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.606686 5426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:36:27.608595 5441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:27.609575 5426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:27.609695 5426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "c113ebc2317546e6a7ad55883caaef53"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "3ef2aee71e218a458240663253bbeffe"
server_key_iv: "3fcb7763097e9e8294f78a6438cde17b"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.609784 5426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:27.627260 5426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:27.629530 5426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:27.629703 5426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:27.630218 5426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:27.630891 5426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:27.630935 5426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:27.630997 5426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:27.631028 5426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:27.638690 5426 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:40747
I20260923 14:36:27.638711 5554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:40747 every 8 connection(s)
I20260923 14:36:27.639442 5426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:27.639643 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 5426
I20260923 14:36:27.639744 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:27.640038 2920 external_mini_cluster.cc:1428] Setting key 14d884cd340ba06fa86a4c187991c5d4
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
Sep 23 14:36:27 dist-test-slave-zl47 krb5kdc[5319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174187, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
I20260923 14:36:27.653785 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:27.641210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:33971 (local address 127.2.218.62:36197)
0923 14:36:27.641331 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:27.641335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:27.642090 (+ 755us) server_negotiation.cc:407] Connection header received
0923 14:36:27.643824 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:27.643827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:27.643875 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:27.643953 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:27.645378 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:27.645901 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:27.647554 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:27.647821 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:27.647909 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:27.650560 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:27.650564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:27.650566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:27.650590 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:27.651849 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:27.652366 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:27.652372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:27.652376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:27.652465 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:27.652773 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:27.652779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:27.652783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:27.653022 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:27.653143 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:27.653526 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 14:36:27.653638 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 14:36:27.654459 5557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:27.641505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36197 (local address 127.2.218.1:33971)
0923 14:36:27.641925 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:27.641974 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:27.643603 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:27.644128 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:27.644138 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:27.644600 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:27.645194 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:27.645217 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:27.646037 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:27.646044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:27.647420 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:27.647434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:27.647977 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:27.647983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:27.648227 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:27.648764 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:27.648791 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:27.650406 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:27.651950 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:27.651958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:27.651971 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:27.652257 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:27.652563 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:27.652565 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:27.652566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:27.652681 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:27.653145 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:27.653149 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:27.653317 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 14:36:27.653466 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 14:36:27.653731 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
I20260923 14:36:27.655593 5555 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36197
I20260923 14:36:27.655813 5555 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:27.656198 5555 heartbeater.cc:511] Master 127.2.218.62:36197 requested a full tablet report, sending...
I20260923 14:36:27.657454 5364 ts_manager.cc:194] Registered new tserver with Master: c113ebc2317546e6a7ad55883caaef53 (127.2.218.1:40747)
I20260923 14:36:27.658440 5364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:33971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:36:27.664968 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36197
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:27.757457 5560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:27.757658 5560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:27.757719 5560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:27.761265 5560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:27.761310 5560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:27.761405 5560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:27.766062 5560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:27.766832 5560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:27.767601 5560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:27.775408 5565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:27.775490 5568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:27.775503 5560 server_base.cc:1034] running on GCE node
W20260923 14:36:27.775408 5566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:27.775981 5560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:27.776564 5560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:27.777807 5560 hybrid_clock.cc:648] HybridClock initialized: now 1790174187777775 us; error 36 us; skew 500 ppm
Sep 23 14:36:27 dist-test-slave-zl47 krb5kdc[5319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174187, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:27.780296 5560 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:36:27.781021 5560 webserver.cc:460] Webserver started at http://127.2.218.2:38017/ using document root <none> and password file <none>
I20260923 14:36:27.781461 5560 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:27.781533 5560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:27.781694 5560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:27.783324 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "c72f2f8b5eb84a359b2be6c0ec648134"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "fb1a19122f7fa019998822449c050745"
server_key_iv: "ae0c00352646e09c4b58428505514969"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.783774 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "c72f2f8b5eb84a359b2be6c0ec648134"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "fb1a19122f7fa019998822449c050745"
server_key_iv: "ae0c00352646e09c4b58428505514969"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.786715 5560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:36:27.788630 5575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:27.789608 5560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:27.789716 5560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "c72f2f8b5eb84a359b2be6c0ec648134"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "fb1a19122f7fa019998822449c050745"
server_key_iv: "ae0c00352646e09c4b58428505514969"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.789801 5560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:27.808414 5560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:27.810652 5560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:27.810803 5560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:27.811233 5560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:27.811854 5560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:27.811914 5560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:27.811965 5560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:27.812000 5560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:27.819645 5560 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:42923
I20260923 14:36:27.819672 5688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:42923 every 8 connection(s)
I20260923 14:36:27.820325 5560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
Sep 23 14:36:27 dist-test-slave-zl47 krb5kdc[5319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174187, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:27.829285 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 5560
I20260923 14:36:27.829380 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:27.829617 2920 external_mini_cluster.cc:1428] Setting key d130333805558a33b3a2086eb62f2d6f
I20260923 14:36:27.832504 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:27.821860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:59287 (local address 127.2.218.62:36197)
0923 14:36:27.821980 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:27.821984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:27.822600 (+ 616us) server_negotiation.cc:407] Connection header received
0923 14:36:27.823746 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:27.823750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:27.823809 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:27.823901 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:27.824956 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:27.825415 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:27.826590 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:27.826787 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:27.826892 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:27.829039 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:27.829048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:27.829050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:27.829091 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:27.830629 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:27.831177 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:27.831188 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:27.831190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:27.831270 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:27.831597 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:27.831600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:27.831601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:27.831776 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:27.831869 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:27.832263 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 14:36:27.832364 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:36:27.833132 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:27.822051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36197 (local address 127.2.218.2:59287)
0923 14:36:27.822464 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:27.822497 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:27.823545 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:27.824041 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:27.824047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:27.824333 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:27.824782 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:27.824795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:27.825535 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:27.825538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:27.826464 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:27.826473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:27.826917 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:27.826921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:27.827084 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:27.827584 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:27.827599 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:27.828889 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:27.830745 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:27.830749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:27.830757 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:27.831054 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:27.831361 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:27.831366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:27.831368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:27.831494 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:27.831867 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:27.831873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:27.832060 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 14:36:27.832222 (+ 162us) client_negotiation.cc:240] Negotiation successful
0923 14:36:27.832460 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260923 14:36:27.834038 5689 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36197
I20260923 14:36:27.834242 5689 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:27.834666 5689 heartbeater.cc:511] Master 127.2.218.62:36197 requested a full tablet report, sending...
I20260923 14:36:27.835573 5364 ts_manager.cc:194] Registered new tserver with Master: c72f2f8b5eb84a359b2be6c0ec648134 (127.2.218.2:42923)
I20260923 14:36:27.836329 5364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:59287
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:36:27.852471 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36197
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:27.945969 5694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:27.946249 5694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:27.946324 5694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:27.949595 5694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:27.949644 5694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:27.949744 5694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:27.953920 5694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:27.954670 5694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:27.955433 5694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:27.963186 5702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:27.963182 5699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:27.963182 5700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:27.963358 5694 server_base.cc:1034] running on GCE node
I20260923 14:36:27.963706 5694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:27.964301 5694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:27.965482 5694 hybrid_clock.cc:648] HybridClock initialized: now 1790174187965467 us; error 26 us; skew 500 ppm
Sep 23 14:36:27 dist-test-slave-zl47 krb5kdc[5319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174187, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:27.967790 5694 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:36:27.968456 5694 webserver.cc:460] Webserver started at http://127.2.218.3:36301/ using document root <none> and password file <none>
I20260923 14:36:27.968845 5694 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:27.968891 5694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:27.969012 5694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:27.970567 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "28c72e47146c47f398cb2b1820c1a220"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "26c5e7812301310654db3d1b12a42aa3"
server_key_iv: "496946192ae18b8e987e4588a98fa8bb"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.970976 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "28c72e47146c47f398cb2b1820c1a220"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "26c5e7812301310654db3d1b12a42aa3"
server_key_iv: "496946192ae18b8e987e4588a98fa8bb"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.973943 5694 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:36:27.975911 5709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:27.976879 5694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:27.976956 5694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "28c72e47146c47f398cb2b1820c1a220"
format_stamp: "Formatted at 2026-09-23 14:36:27 on dist-test-slave-zl47"
server_key: "26c5e7812301310654db3d1b12a42aa3"
server_key_iv: "496946192ae18b8e987e4588a98fa8bb"
server_key_version: "encryptionkey@0"
I20260923 14:36:27.977020 5694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:27.987520 5694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:27.989967 5694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:27.990088 5694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:27.990525 5694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:27.991103 5694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:27.991142 5694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:27.991171 5694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:27.991184 5694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:27.998618 5694 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:35953
I20260923 14:36:27.998641 5822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:35953 every 8 connection(s)
I20260923 14:36:27.999346 5694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:36:28.006685 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 5694
I20260923 14:36:28.006757 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
Sep 23 14:36:28 dist-test-slave-zl47 krb5kdc[5319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174187, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:28.006968 2920 external_mini_cluster.cc:1428] Setting key 0cefcdab092b1b2c7ef11731388e0089
I20260923 14:36:28.010766 5404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:28.000748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:39389 (local address 127.2.218.62:36197)
0923 14:36:28.000893 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:28.000897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:28.001507 (+ 610us) server_negotiation.cc:407] Connection header received
0923 14:36:28.002688 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:28.002691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:28.002734 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:28.002799 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:28.003692 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:28.004126 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:28.005057 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:28.005250 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:28.005319 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:28.007483 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:28.007485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:28.007487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:28.007510 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:28.009049 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:28.009464 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:28.009468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:28.009469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:28.009542 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:28.009835 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:28.009839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:28.009841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:28.010015 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:28.010136 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:28.010519 (+ 383us) server_negotiation.cc:299] Negotiation successful
0923 14:36:28.010628 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:36:28.011325 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:28.001027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36197 (local address 127.2.218.3:39389)
0923 14:36:28.001384 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:28.001418 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:28.002512 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:28.002929 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:28.002934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:28.003218 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:28.003571 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:28.003593 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:28.004212 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:28.004214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:28.004963 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:28.004971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:28.005360 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:28.005362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:28.005475 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:28.005983 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:28.005995 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:28.007310 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:28.009150 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:28.009154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:28.009164 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:28.009373 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:28.009636 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:28.009638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:28.009640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:28.009752 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:28.010143 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:28.010150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:28.010332 (+ 182us) client_negotiation.cc:769] Sending connection context
0923 14:36:28.010485 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 14:36:28.010721 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
I20260923 14:36:28.012212 5823 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36197
I20260923 14:36:28.012406 5823 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:28.012777 5823 heartbeater.cc:511] Master 127.2.218.62:36197 requested a full tablet report, sending...
I20260923 14:36:28.013605 5364 ts_manager.cc:194] Registered new tserver with Master: 28c72e47146c47f398cb2b1820c1a220 (127.2.218.3:35953)
I20260923 14:36:28.014076 5364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:39389
I20260923 14:36:28.020270 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 14:36:28.030897 5404 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:28.029735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.2.218.62:36197)
0923 14:36:28.029836 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:28.029839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:28.030003 (+ 164us) server_negotiation.cc:407] Connection header received
0923 14:36:28.030085 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:28.030087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:28.030167 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:28.030228 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:28.030771 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47176: BlockingRecv error: recv got EOF from 127.0.0.1:47176 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:36:28.031827 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 5426
I20260923 14:36:28.036643 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 5560
I20260923 14:36:28.043108 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 5694
I20260923 14:36:28.047817 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 5332
2026-09-23T14:36:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1807 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:28 dist-test-slave-zl47 krb5kdc[5835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:28 dist-test-slave-zl47 krb5kdc[5835](info): set up 2 sockets
Sep 23 14:36:28 dist-test-slave-zl47 krb5kdc[5835](info): commencing operation
krb5kdc: starting...
W20260923 14:36:28.953961 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.886s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:29 dist-test-slave-zl47 krb5kdc[5835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:29.052759 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:40705
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:45481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:40705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:29.145571 5848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:29.145753 5848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:29.145785 5848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:29.149183 5848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:29.149227 5848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:29.149243 5848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:29.149291 5848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:29.149313 5848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:29.153467 5848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:40705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:40705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:29.154245 5848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:29.154994 5848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:29.161834 5854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:29.161859 5853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:29.161896 5856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:29.161994 5848 server_base.cc:1034] running on GCE node
I20260923 14:36:29.162567 5848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:29.163542 5848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:29.164724 5848 hybrid_clock.cc:648] HybridClock initialized: now 1790174189164699 us; error 37 us; skew 500 ppm
Sep 23 14:36:29 dist-test-slave-zl47 krb5kdc[5835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174189, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:29.171972 5848 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:29.172684 5848 webserver.cc:460] Webserver started at http://127.2.218.62:42487/ using document root <none> and password file <none>
I20260923 14:36:29.173188 5848 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:29.173274 5848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:29.173482 5848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:29.175212 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "e06f5d11aea348f2a4aba05df26faa0a"
format_stamp: "Formatted at 2026-09-23 14:36:29 on dist-test-slave-zl47"
server_key: "8b8c5556a1a56f91969c1eb37dd81e0f"
server_key_iv: "1b94fce0a25904ff2b85d709458d757e"
server_key_version: "encryptionkey@0"
I20260923 14:36:29.175676 5848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "e06f5d11aea348f2a4aba05df26faa0a"
format_stamp: "Formatted at 2026-09-23 14:36:29 on dist-test-slave-zl47"
server_key: "8b8c5556a1a56f91969c1eb37dd81e0f"
server_key_iv: "1b94fce0a25904ff2b85d709458d757e"
server_key_version: "encryptionkey@0"
I20260923 14:36:29.178819 5848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260923 14:36:29.180807 5863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:29.181809 5848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:29.181926 5848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "e06f5d11aea348f2a4aba05df26faa0a"
format_stamp: "Formatted at 2026-09-23 14:36:29 on dist-test-slave-zl47"
server_key: "8b8c5556a1a56f91969c1eb37dd81e0f"
server_key_iv: "1b94fce0a25904ff2b85d709458d757e"
server_key_version: "encryptionkey@0"
I20260923 14:36:29.182019 5848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:29.195386 5848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:29.197630 5848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:29.197789 5848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:29.203409 5848 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:40705
I20260923 14:36:29.203646 5915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:40705 every 8 connection(s)
I20260923 14:36:29.204155 5848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:29.206425 5916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:29.207149 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 5848
I20260923 14:36:29.207232 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:29.207471 2920 external_mini_cluster.cc:1428] Setting key a1a67f7c8b8f45bbbcb6349957f23425
I20260923 14:36:29.212061 5916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a: Bootstrap starting.
I20260923 14:36:29.214260 5916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:29.214883 5916 log.cc:824] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a: Log is configured to *not* fsync() on all Append() calls
Sep 23 14:36:29 dist-test-slave-zl47 krb5kdc[5835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:29.216753 5916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a: No bootstrap required, opened a new log
I20260923 14:36:29.220309 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:29.208569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.2.218.62:40705)
0923 14:36:29.208978 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:29.208995 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:29.209031 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:36:29.209760 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:29.209780 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:29.210136 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:29.210440 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:29.210984 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:29.212224 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:29.212957 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:29.213544 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:29.213694 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:29.215805 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:29.215810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:29.215823 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:29.215848 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:29.217639 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:29.218039 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:29.218045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:29.218050 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:29.218236 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:29.218473 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:29.218476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:29.218478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:29.218937 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:29.219162 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:29.219304 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 14:36:29.219544 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":93,"threads_started":1}
I20260923 14:36:29.220887 5916 raft_consensus.cc:348] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06f5d11aea348f2a4aba05df26faa0a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 40705 } }
I20260923 14:36:29.221127 5916 raft_consensus.cc:374] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:29.221200 5916 raft_consensus.cc:729] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06f5d11aea348f2a4aba05df26faa0a, State: Initialized, Role: FOLLOWER
I20260923 14:36:29.221599 5916 consensus_queue.cc:260] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06f5d11aea348f2a4aba05df26faa0a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 40705 } }
I20260923 14:36:29.221716 5916 raft_consensus.cc:388] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:29.221805 5916 raft_consensus.cc:482] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:29.221908 5916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:29.222961 5916 raft_consensus.cc:504] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06f5d11aea348f2a4aba05df26faa0a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 40705 } }
I20260923 14:36:29.223122 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:36:29.223268 5916 leader_election.cc:302] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e06f5d11aea348f2a4aba05df26faa0a; no voters:
I20260923 14:36:29.223536 5916 leader_election.cc:288] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:29.223631 5921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:29.223819 5921 raft_consensus.cc:686] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [term 1 LEADER]: Becoming Leader. State: Replica: e06f5d11aea348f2a4aba05df26faa0a, State: Running, Role: LEADER
I20260923 14:36:29.224061 5921 consensus_queue.cc:237] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06f5d11aea348f2a4aba05df26faa0a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 40705 } }
I20260923 14:36:29.224682 5916 sys_catalog.cc:564] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:29.224905 5923 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e06f5d11aea348f2a4aba05df26faa0a. Latest consensus state: current_term: 1 leader_uuid: "e06f5d11aea348f2a4aba05df26faa0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06f5d11aea348f2a4aba05df26faa0a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 40705 } } }
I20260923 14:36:29.225016 5923 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:29.225244 5922 sys_catalog.cc:455] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e06f5d11aea348f2a4aba05df26faa0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e06f5d11aea348f2a4aba05df26faa0a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 40705 } } }
I20260923 14:36:29.225310 5922 sys_catalog.cc:458] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:29.225409 5926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:29.228161 5926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:29.233150 5926 catalog_manager.cc:1347] Generated new cluster ID: c1fa74a0ece747a6a851e546064ca94e
I20260923 14:36:29.233212 5926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:29.247253 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:29.242744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.2.218.62:40705)
0923 14:36:29.242827 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:29.242830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:29.243050 (+ 220us) server_negotiation.cc:407] Connection header received
0923 14:36:29.243133 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:29.243136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:29.243179 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:29.243250 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:29.243686 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:29.244332 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:29.245271 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:29.245594 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:29.245683 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:29.246419 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:29.246423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:29.246426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:29.246857 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:29.246886 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:29.246904 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:29.246986 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:29.247081 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 14:36:29.247131 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 14:36:29.251996 5926 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:29.252661 5926 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:29.262529 5926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e06f5d11aea348f2a4aba05df26faa0a: Generated new TSK 0
I20260923 14:36:29.262991 5926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:36:29.272506 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 5848
2026-09-23T14:36:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1222 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:29 dist-test-slave-zl47 krb5kdc[5949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:29 dist-test-slave-zl47 krb5kdc[5949](info): set up 2 sockets
Sep 23 14:36:29 dist-test-slave-zl47 krb5kdc[5949](info): commencing operation
krb5kdc: starting...
W20260923 14:36:30.178016 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.888s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:30 dist-test-slave-zl47 krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:30.276676 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:34633
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:34633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:30.366798 5962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:30.366976 5962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:30.367008 5962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:30.370385 5962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:30.370424 5962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:30.370441 5962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:30.370455 5962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:30.370493 5962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:30.374919 5962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:34633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:34633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:30.375634 5962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:30.376362 5962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:30.383009 5968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:30.383029 5967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:30.383021 5970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:30.383086 5962 server_base.cc:1034] running on GCE node
I20260923 14:36:30.383616 5962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:30.384374 5962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:30.385521 5962 hybrid_clock.cc:648] HybridClock initialized: now 1790174190385505 us; error 25 us; skew 500 ppm
Sep 23 14:36:30 dist-test-slave-zl47 krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174190, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:30.387704 5962 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:30.388355 5962 webserver.cc:460] Webserver started at http://127.2.218.62:41713/ using document root <none> and password file <none>
I20260923 14:36:30.388783 5962 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:30.388859 5962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:30.389037 5962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:30.390674 5962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "734039456a8f459c952aa89f71965d0f"
format_stamp: "Formatted at 2026-09-23 14:36:30 on dist-test-slave-zl47"
server_key: "e4778cfbf7836d574268846d03d2bcac"
server_key_iv: "ba447f1d88d5f6c32533d7f95519c1ea"
server_key_version: "encryptionkey@0"
I20260923 14:36:30.391175 5962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "734039456a8f459c952aa89f71965d0f"
format_stamp: "Formatted at 2026-09-23 14:36:30 on dist-test-slave-zl47"
server_key: "e4778cfbf7836d574268846d03d2bcac"
server_key_iv: "ba447f1d88d5f6c32533d7f95519c1ea"
server_key_version: "encryptionkey@0"
I20260923 14:36:30.393983 5962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:36:30.395949 5977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:30.396883 5962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:30.397001 5962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "734039456a8f459c952aa89f71965d0f"
format_stamp: "Formatted at 2026-09-23 14:36:30 on dist-test-slave-zl47"
server_key: "e4778cfbf7836d574268846d03d2bcac"
server_key_iv: "ba447f1d88d5f6c32533d7f95519c1ea"
server_key_version: "encryptionkey@0"
I20260923 14:36:30.397089 5962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:30.417382 5962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:30.419783 5962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:30.419965 5962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:30.427124 5962 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:34633
I20260923 14:36:30.427296 6029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:34633 every 8 connection(s)
I20260923 14:36:30.427938 5962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:30.430187 6030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:30.431381 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 5962
I20260923 14:36:30.431504 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:30.431733 2920 external_mini_cluster.cc:1428] Setting key ce5da6d1dda9477d6842ae4729f89686
I20260923 14:36:30.435101 6030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f: Bootstrap starting.
I20260923 14:36:30.436834 6030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:30.437430 6030 log.cc:824] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:30.439213 6030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f: No bootstrap required, opened a new log
Sep 23 14:36:30 dist-test-slave-zl47 krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:30.441761 6030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734039456a8f459c952aa89f71965d0f" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34633 } }
I20260923 14:36:30.441921 6030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:30.441948 6030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734039456a8f459c952aa89f71965d0f, State: Initialized, Role: FOLLOWER
I20260923 14:36:30.442412 6030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734039456a8f459c952aa89f71965d0f" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34633 } }
I20260923 14:36:30.442550 6030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:30.442606 6030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:30.442687 6030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:30.443408 6030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734039456a8f459c952aa89f71965d0f" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34633 } }
I20260923 14:36:30.443710 6030 leader_election.cc:302] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 734039456a8f459c952aa89f71965d0f; no voters:
I20260923 14:36:30.443977 6030 leader_election.cc:288] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:30.444036 6035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:30.444254 6035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [term 1 LEADER]: Becoming Leader. State: Replica: 734039456a8f459c952aa89f71965d0f, State: Running, Role: LEADER
I20260923 14:36:30.444490 6035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734039456a8f459c952aa89f71965d0f" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34633 } }
I20260923 14:36:30.444629 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:30.432901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44484 (local address 127.2.218.62:34633)
0923 14:36:30.433265 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:30.433278 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:30.433307 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:36:30.433840 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:30.433853 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:30.434168 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:30.434457 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:30.435220 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:30.436032 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:30.436810 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:30.437275 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:30.437436 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:30.439811 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:30.439815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:30.439825 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:30.439848 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:30.441616 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:30.442050 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:30.442057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:30.442070 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:30.442220 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:30.442513 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:30.442517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:30.442519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:30.442954 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:30.443179 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:30.443456 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 14:36:30.443736 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260923 14:36:30.445050 6030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:30.445587 6035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "734039456a8f459c952aa89f71965d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734039456a8f459c952aa89f71965d0f" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34633 } } }
I20260923 14:36:30.445699 6035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:30.446029 6036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 734039456a8f459c952aa89f71965d0f. Latest consensus state: current_term: 1 leader_uuid: "734039456a8f459c952aa89f71965d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "734039456a8f459c952aa89f71965d0f" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34633 } } }
I20260923 14:36:30.446147 6036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:30.446542 6039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:30.449177 6039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:30.457368 6039 catalog_manager.cc:1347] Generated new cluster ID: 0e243c998e184fe49444d4191cfd9fd5
I20260923 14:36:30.457436 6039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:30.474514 6039 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:30.475247 6039 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:30.481153 6039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 734039456a8f459c952aa89f71965d0f: Generated new TSK 0
I20260923 14:36:30.481696 6039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:36:30.514467 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34633
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:30.632818 6056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:30.632997 6056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:30.633030 6056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:30.636651 6056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:30.636693 6056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:30.636785 6056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:30.641028 6056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:30.641742 6056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:30.642563 6056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:30.650233 6061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:30.650262 6064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:30.650309 6062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:30.650346 6056 server_base.cc:1034] running on GCE node
I20260923 14:36:30.650707 6056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:30.651216 6056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:30.652395 6056 hybrid_clock.cc:648] HybridClock initialized: now 1790174190652369 us; error 42 us; skew 500 ppm
Sep 23 14:36:30 dist-test-slave-zl47 krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174190, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:30.654659 6056 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:36:30.655345 6056 webserver.cc:460] Webserver started at http://127.2.218.1:41859/ using document root <none> and password file <none>
I20260923 14:36:30.655771 6056 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:30.655820 6056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:30.656023 6056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:30.657572 6056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "280f0957c43d434a91c45c4035ee97ed"
format_stamp: "Formatted at 2026-09-23 14:36:30 on dist-test-slave-zl47"
server_key: "cfb88528bc20b8e7c52e0ac667b76734"
server_key_iv: "97e0c415db0afb6f13e81e486e140250"
server_key_version: "encryptionkey@0"
I20260923 14:36:30.658013 6056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "280f0957c43d434a91c45c4035ee97ed"
format_stamp: "Formatted at 2026-09-23 14:36:30 on dist-test-slave-zl47"
server_key: "cfb88528bc20b8e7c52e0ac667b76734"
server_key_iv: "97e0c415db0afb6f13e81e486e140250"
server_key_version: "encryptionkey@0"
I20260923 14:36:30.660857 6056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:36:30.662770 6071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:30.663671 6056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:30.663789 6056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "280f0957c43d434a91c45c4035ee97ed"
format_stamp: "Formatted at 2026-09-23 14:36:30 on dist-test-slave-zl47"
server_key: "cfb88528bc20b8e7c52e0ac667b76734"
server_key_iv: "97e0c415db0afb6f13e81e486e140250"
server_key_version: "encryptionkey@0"
I20260923 14:36:30.663882 6056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:30.680565 6056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:30.683177 6056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:30.683344 6056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:30.683815 6056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:30.684429 6056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:30.684490 6056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:30.684545 6056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:30.684597 6056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:30.692008 6056 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:38935
I20260923 14:36:30.692042 6184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:38935 every 8 connection(s)
I20260923 14:36:30.692653 6056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:30.699599 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 6056
I20260923 14:36:30.699718 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:30.699980 2920 external_mini_cluster.cc:1428] Setting key e592af02960a92cdef0420ec4d9d4d1e
Sep 23 14:36:30 dist-test-slave-zl47 krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174190, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:30.704322 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:30.693967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:38795 (local address 127.2.218.62:34633)
0923 14:36:30.694080 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:30.694083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:30.694666 (+ 583us) server_negotiation.cc:407] Connection header received
0923 14:36:30.695802 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:30.695805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:30.695868 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:30.695958 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:30.696865 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:30.697327 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:30.698311 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:30.698529 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:30.698637 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:30.700902 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:30.700905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:30.700907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:30.700935 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:30.702534 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:30.702997 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:30.703002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:30.703003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:30.703082 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:30.703399 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:30.703404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:30.703405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:30.703569 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:30.703686 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:30.704081 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 14:36:30.704182 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:36:30.704885 6187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:30.694193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:34633 (local address 127.2.218.1:38795)
0923 14:36:30.694539 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:30.694573 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:30.695620 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:30.696104 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:30.696110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:30.696378 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:30.696731 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:30.696741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:30.697440 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:30.697443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:30.698201 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:30.698209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:30.698667 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:30.698671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:30.698808 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:30.699288 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:30.699301 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:30.700745 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:30.702655 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:30.702659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:30.702667 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:30.702874 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:30.703177 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:30.703183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:30.703186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:30.703305 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:30.703686 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:30.703693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:30.703877 (+ 184us) client_negotiation.cc:769] Sending connection context
0923 14:36:30.704045 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 14:36:30.704286 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260923 14:36:30.705966 6185 heartbeater.cc:347] Connected to a master server at 127.2.218.62:34633
I20260923 14:36:30.706207 6185 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:30.706571 6185 heartbeater.cc:511] Master 127.2.218.62:34633 requested a full tablet report, sending...
I20260923 14:36:30.707779 5994 ts_manager.cc:194] Registered new tserver with Master: 280f0957c43d434a91c45c4035ee97ed (127.2.218.1:38935)
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
I20260923 14:36:30.708787 5994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:38795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:36:30.724483 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34633
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:30.817401 6190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:30.817596 6190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:30.817636 6190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:30.821293 6190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:30.821342 6190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:30.821424 6190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:30.825824 6190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:30.826638 6190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:30.827427 6190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:30.835557 6196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:30.835600 6190 server_base.cc:1034] running on GCE node
W20260923 14:36:30.835561 6195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:30.835570 6198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:30.836167 6190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:30.836612 6190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:30.837822 6190 hybrid_clock.cc:648] HybridClock initialized: now 1790174190837785 us; error 36 us; skew 500 ppm
Sep 23 14:36:30 dist-test-slave-zl47 krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174190, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:30.840333 6190 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:36:30.841025 6190 webserver.cc:460] Webserver started at http://127.2.218.2:41173/ using document root <none> and password file <none>
I20260923 14:36:30.841473 6190 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:30.841554 6190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:30.841744 6190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:30.843360 6190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "1eed910baaa64f11adf978dc258125a5"
format_stamp: "Formatted at 2026-09-23 14:36:30 on dist-test-slave-zl47"
server_key: "174f7ae83c6f49fed9ccf1160bc790e7"
server_key_iv: "d0472e7fa342e7178e35bea34c8855f7"
server_key_version: "encryptionkey@0"
I20260923 14:36:30.843806 6190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "1eed910baaa64f11adf978dc258125a5"
format_stamp: "Formatted at 2026-09-23 14:36:30 on dist-test-slave-zl47"
server_key: "174f7ae83c6f49fed9ccf1160bc790e7"
server_key_iv: "d0472e7fa342e7178e35bea34c8855f7"
server_key_version: "encryptionkey@0"
I20260923 14:36:30.846968 6190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260923 14:36:30.848870 6205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:30.849843 6190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:30.849967 6190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "1eed910baaa64f11adf978dc258125a5"
format_stamp: "Formatted at 2026-09-23 14:36:30 on dist-test-slave-zl47"
server_key: "174f7ae83c6f49fed9ccf1160bc790e7"
server_key_iv: "d0472e7fa342e7178e35bea34c8855f7"
server_key_version: "encryptionkey@0"
I20260923 14:36:30.850059 6190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:30.858485 6190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:30.861044 6190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:30.861207 6190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:30.861635 6190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:30.862286 6190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:30.862346 6190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:30.862396 6190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:30.862438 6190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:30.870185 6190 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:36211
I20260923 14:36:30.870229 6318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:36211 every 8 connection(s)
I20260923 14:36:30.870844 6190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
Sep 23 14:36:30 dist-test-slave-zl47 krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174190, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:30.878782 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 6190
I20260923 14:36:30.878901 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:30.879170 2920 external_mini_cluster.cc:1428] Setting key 3d6550c2164563d4f3e6db3c21edbacd
I20260923 14:36:30.881654 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:30.872282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:38829 (local address 127.2.218.62:34633)
0923 14:36:30.872389 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:30.872393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:30.873012 (+ 619us) server_negotiation.cc:407] Connection header received
0923 14:36:30.874077 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:30.874080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:30.874165 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:30.874231 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:30.875064 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:30.875533 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:30.876452 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:30.876642 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:30.876711 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:30.878616 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:30.878621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:30.878624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:30.878649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:30.879928 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:30.880363 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:30.880368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:30.880369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:30.880438 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:30.880731 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:30.880735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:30.880737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:30.880898 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:30.880989 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:30.881384 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 14:36:30.881506 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:36:30.882508 6321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:30.872542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:34633 (local address 127.2.218.2:38829)
0923 14:36:30.872884 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:30.872915 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:30.873909 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:30.874342 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:30.874347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:30.874616 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:30.874945 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:30.874954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:30.875633 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:30.875635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:30.876356 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:30.876363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:30.876744 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:30.876747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:30.876852 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:30.877288 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:30.877301 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:30.878472 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:30.880033 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:30.880038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:30.880049 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:30.880268 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:30.880520 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:30.880524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:30.880525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:30.880645 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:30.881000 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:30.881007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:30.881193 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 14:36:30.881344 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 14:36:30.881616 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
I20260923 14:36:30.883438 6319 heartbeater.cc:347] Connected to a master server at 127.2.218.62:34633
I20260923 14:36:30.883621 6319 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:30.883970 6319 heartbeater.cc:511] Master 127.2.218.62:34633 requested a full tablet report, sending...
I20260923 14:36:30.884856 5994 ts_manager.cc:194] Registered new tserver with Master: 1eed910baaa64f11adf978dc258125a5 (127.2.218.2:36211)
I20260923 14:36:30.885339 5994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:38829
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:36:30.901993 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34633
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:30.993891 6324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:30.994119 6324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:30.994189 6324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:30.997484 6324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:30.997550 6324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:30.997623 6324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:31.002024 6324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:31.002805 6324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:31.003635 6324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:31.011744 6332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:31.011754 6329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:31.011754 6330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:31.011941 6324 server_base.cc:1034] running on GCE node
I20260923 14:36:31.012344 6324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:31.012825 6324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:31.014019 6324 hybrid_clock.cc:648] HybridClock initialized: now 1790174191013994 us; error 39 us; skew 500 ppm
Sep 23 14:36:31 dist-test-slave-zl47 krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174191, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:31.016631 6324 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:36:31.017298 6324 webserver.cc:460] Webserver started at http://127.2.218.3:38123/ using document root <none> and password file <none>
I20260923 14:36:31.017750 6324 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:31.017827 6324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:31.018011 6324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:31.019769 6324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "74d8308fd21c422c94d624d8ecf0f977"
format_stamp: "Formatted at 2026-09-23 14:36:31 on dist-test-slave-zl47"
server_key: "afa73eb8d8e8469a5320bc097a980bee"
server_key_iv: "5f9d59e1e878bdaeb4cefaa5ade9c436"
server_key_version: "encryptionkey@0"
I20260923 14:36:31.020234 6324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "74d8308fd21c422c94d624d8ecf0f977"
format_stamp: "Formatted at 2026-09-23 14:36:31 on dist-test-slave-zl47"
server_key: "afa73eb8d8e8469a5320bc097a980bee"
server_key_iv: "5f9d59e1e878bdaeb4cefaa5ade9c436"
server_key_version: "encryptionkey@0"
I20260923 14:36:31.023213 6324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:36:31.025065 6339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:31.026041 6324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:36:31.026188 6324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "74d8308fd21c422c94d624d8ecf0f977"
format_stamp: "Formatted at 2026-09-23 14:36:31 on dist-test-slave-zl47"
server_key: "afa73eb8d8e8469a5320bc097a980bee"
server_key_iv: "5f9d59e1e878bdaeb4cefaa5ade9c436"
server_key_version: "encryptionkey@0"
I20260923 14:36:31.026283 6324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:31.034746 6324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:31.037249 6324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:31.037411 6324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:31.037858 6324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:31.038544 6324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:31.038606 6324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:31.038686 6324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:31.038729 6324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:31.045984 6324 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:41627
I20260923 14:36:31.046005 6452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:41627 every 8 connection(s)
I20260923 14:36:31.046768 6324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
Sep 23 14:36:31 dist-test-slave-zl47 krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174191, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:31.056389 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 6324
I20260923 14:36:31.056504 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:31.056720 2920 external_mini_cluster.cc:1428] Setting key 858d1492f2c26cb0790a962350b221c4
I20260923 14:36:31.057866 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:31.048201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:55841 (local address 127.2.218.62:34633)
0923 14:36:31.048318 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:31.048321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:31.048932 (+ 611us) server_negotiation.cc:407] Connection header received
0923 14:36:31.050083 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:31.050110 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:31.050163 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:31.050252 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:31.051167 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:31.051674 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:31.052723 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:31.052980 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:31.053093 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:31.054932 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:31.054939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:31.054941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:31.054961 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:31.056027 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:31.056462 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:31.056466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:31.056468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:31.056564 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:31.056874 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:31.056877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:31.056878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:31.057098 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:31.057178 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:31.057624 (+ 446us) server_negotiation.cc:299] Negotiation successful
0923 14:36:31.057725 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:36:31.058499 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:31.048463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:34633 (local address 127.2.218.3:55841)
0923 14:36:31.048808 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:31.048841 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:31.049907 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:31.050382 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:31.050387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:31.050667 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:31.051023 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:31.051045 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:31.051790 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:31.051797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:31.052602 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:31.052609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:31.053112 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:31.053117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:31.053255 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:31.053626 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:31.053639 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:31.054793 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:31.056146 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:31.056150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:31.056161 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:31.056371 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:31.056638 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:31.056641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:31.056643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:31.056754 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:31.057177 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:31.057182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:31.057408 (+ 226us) client_negotiation.cc:769] Sending connection context
0923 14:36:31.057571 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 14:36:31.057822 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":94,"threads_started":1}
I20260923 14:36:31.059458 6453 heartbeater.cc:347] Connected to a master server at 127.2.218.62:34633
I20260923 14:36:31.059671 6453 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:31.060103 6453 heartbeater.cc:511] Master 127.2.218.62:34633 requested a full tablet report, sending...
I20260923 14:36:31.060895 5994 ts_manager.cc:194] Registered new tserver with Master: 74d8308fd21c422c94d624d8ecf0f977 (127.2.218.3:41627)
I20260923 14:36:31.061463 5994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:55841
I20260923 14:36:31.069847 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 14:36:31 dist-test-slave-zl47 krb5kdc[5949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174191, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 14:36:31 dist-test-slave-zl47 krb5kdc[5949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174191, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:31.090999 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:31.084141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44498 (local address 127.2.218.62:34633)
0923 14:36:31.084246 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:31.084249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:31.084404 (+ 155us) server_negotiation.cc:407] Connection header received
0923 14:36:31.084484 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:31.084486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:31.084518 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:31.084589 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:31.085124 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:31.085540 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:31.086278 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:31.086452 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:31.086526 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:31.088421 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:31.088424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:31.088425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:31.088445 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:31.089708 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:31.090088 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:31.090123 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:31.090127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:31.090204 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:31.090411 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:31.090413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:31.090414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:31.090592 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:31.090687 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:31.090832 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 14:36:31.090907 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 14:36:31.091341 5994 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44498
I20260923 14:36:31.092084 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 6056
I20260923 14:36:31.096818 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 6190
I20260923 14:36:31.101195 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 6324
I20260923 14:36:31.105489 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 5962
2026-09-23T14:36:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1835 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:31 dist-test-slave-zl47 krb5kdc[6465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:31 dist-test-slave-zl47 krb5kdc[6465](info): set up 2 sockets
Sep 23 14:36:31 dist-test-slave-zl47 krb5kdc[6465](info): commencing operation
krb5kdc: starting...
W20260923 14:36:32.022372 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.897s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:32.124912 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:37149
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:37149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:32.220021 6478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:32.220227 6478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:32.220263 6478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:32.223695 6478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:32.223742 6478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:32.223762 6478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:32.223778 6478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:32.223793 6478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:32.228113 6478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:37149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:37149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:32.228834 6478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:32.229596 6478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:32.236791 6486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:32.236819 6484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:32.236791 6483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:32.237021 6478 server_base.cc:1034] running on GCE node
I20260923 14:36:32.237547 6478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:32.238471 6478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:32.239622 6478 hybrid_clock.cc:648] HybridClock initialized: now 1790174192239589 us; error 43 us; skew 500 ppm
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:32.242035 6478 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:32.242834 6478 webserver.cc:460] Webserver started at http://127.2.218.62:36049/ using document root <none> and password file <none>
I20260923 14:36:32.243299 6478 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:32.243355 6478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:32.243584 6478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:32.245232 6478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "be967385e3164480a0c26f862b6c659a"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "cd2ae31ba6c21e673c6d0f15fa0b624f"
server_key_iv: "da4dd2ab756de20ffa37ae6e4511e59e"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.245740 6478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "be967385e3164480a0c26f862b6c659a"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "cd2ae31ba6c21e673c6d0f15fa0b624f"
server_key_iv: "da4dd2ab756de20ffa37ae6e4511e59e"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.248935 6478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260923 14:36:32.250977 6493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:32.251964 6478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:32.252081 6478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "be967385e3164480a0c26f862b6c659a"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "cd2ae31ba6c21e673c6d0f15fa0b624f"
server_key_iv: "da4dd2ab756de20ffa37ae6e4511e59e"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.252177 6478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:32.260797 6478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:32.263479 6478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:32.263620 6478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:32.269647 6478 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:37149
I20260923 14:36:32.269656 6545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:37149 every 8 connection(s)
I20260923 14:36:32.270522 6478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:32.272852 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:32.277143 6546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a: Bootstrap starting.
I20260923 14:36:32.279122 6546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:32.279425 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 6478
I20260923 14:36:32.279546 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:32.279788 2920 external_mini_cluster.cc:1428] Setting key e700c9318ce8344d1647253fd0214865
I20260923 14:36:32.279822 6546 log.cc:824] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:32.281659 6546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a: No bootstrap required, opened a new log
I20260923 14:36:32.285349 6546 raft_consensus.cc:348] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be967385e3164480a0c26f862b6c659a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37149 } }
I20260923 14:36:32.285591 6546 raft_consensus.cc:374] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:32.285658 6546 raft_consensus.cc:729] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be967385e3164480a0c26f862b6c659a, State: Initialized, Role: FOLLOWER
I20260923 14:36:32.286063 6546 consensus_queue.cc:260] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be967385e3164480a0c26f862b6c659a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37149 } }
I20260923 14:36:32.286230 6546 raft_consensus.cc:388] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:32.286309 6546 raft_consensus.cc:482] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:32.286397 6546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:32.287142 6546 raft_consensus.cc:504] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be967385e3164480a0c26f862b6c659a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37149 } }
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:32.287459 6546 leader_election.cc:302] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be967385e3164480a0c26f862b6c659a; no voters:
I20260923 14:36:32.287755 6546 leader_election.cc:288] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:32.287858 6551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:32.288053 6551 raft_consensus.cc:686] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [term 1 LEADER]: Becoming Leader. State: Replica: be967385e3164480a0c26f862b6c659a, State: Running, Role: LEADER
I20260923 14:36:32.288367 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be967385e3164480a0c26f862b6c659a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37149 } }
I20260923 14:36:32.288985 6546 sys_catalog.cc:564] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:32.290547 6553 sys_catalog.cc:455] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [sys.catalog]: SysCatalogTable state changed. Reason: New leader be967385e3164480a0c26f862b6c659a. Latest consensus state: current_term: 1 leader_uuid: "be967385e3164480a0c26f862b6c659a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be967385e3164480a0c26f862b6c659a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37149 } } }
I20260923 14:36:32.290664 6553 sys_catalog.cc:458] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:32.291036 6566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:32.291692 6552 sys_catalog.cc:455] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be967385e3164480a0c26f862b6c659a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be967385e3164480a0c26f862b6c659a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37149 } } }
I20260923 14:36:32.291819 6552 sys_catalog.cc:458] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:32.292861 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:32.280750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.2.218.62:37149)
0923 14:36:32.281097 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:32.281110 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:32.281140 (+ 30us) server_negotiation.cc:407] Connection header received
0923 14:36:32.281654 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:32.281676 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:32.281991 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:32.282274 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:32.283124 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.283889 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.285011 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.285381 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.285474 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:32.287986 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:32.287992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:32.288008 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:32.288039 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:32.290137 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.290632 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.290637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.290644 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.290799 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.291089 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.291093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.291095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.291478 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:32.291695 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:32.292059 (+ 364us) server_negotiation.cc:299] Negotiation successful
0923 14:36:32.292287 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":105,"threads_started":1}
I20260923 14:36:32.294706 6566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:32.299317 6566 catalog_manager.cc:1347] Generated new cluster ID: 1993c0787c6c4dfe8a8713ed27382e10
I20260923 14:36:32.299379 6566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:32.311620 6566 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:32.312319 6566 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:32.319389 6566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be967385e3164480a0c26f862b6c659a: Generated new TSK 0
I20260923 14:36:32.319885 6566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:36:32.346644 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37149
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:32.444442 6572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:32.444630 6572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:32.444702 6572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:32.448143 6572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:32.448192 6572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:32.448290 6572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:32.453090 6572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:32.453852 6572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:32.454764 6572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:32.462797 6580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:32.462883 6572 server_base.cc:1034] running on GCE node
W20260923 14:36:32.462826 6578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:32.462841 6577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:32.463310 6572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:32.463817 6572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:32.464986 6572 hybrid_clock.cc:648] HybridClock initialized: now 1790174192464960 us; error 36 us; skew 500 ppm
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:32.467351 6572 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:36:32.468060 6572 webserver.cc:460] Webserver started at http://127.2.218.1:35263/ using document root <none> and password file <none>
I20260923 14:36:32.468529 6572 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:32.468607 6572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:32.468804 6572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:32.470541 6572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "3b653532852d44b3b9d7aedbaf31964a"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "c0e6f7d05a080ed838952ba9af1afce0"
server_key_iv: "cd6cacdad9915223a0dde0c6c61437e9"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.471001 6572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "3b653532852d44b3b9d7aedbaf31964a"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "c0e6f7d05a080ed838952ba9af1afce0"
server_key_iv: "cd6cacdad9915223a0dde0c6c61437e9"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.474222 6572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:36:32.476182 6587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:32.477193 6572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:32.477303 6572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "3b653532852d44b3b9d7aedbaf31964a"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "c0e6f7d05a080ed838952ba9af1afce0"
server_key_iv: "cd6cacdad9915223a0dde0c6c61437e9"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.477391 6572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:32.486052 6572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:32.488300 6572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:32.488464 6572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:32.488924 6572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:32.489637 6572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:32.489740 6572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:32.489796 6572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:32.489823 6572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:32.497411 6572 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:37631
I20260923 14:36:32.497430 6700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:37631 every 8 connection(s)
I20260923 14:36:32.498083 6572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:32.501427 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 6572
I20260923 14:36:32.501534 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:32.501789 2920 external_mini_cluster.cc:1428] Setting key eaccddfa702224f212bf01838530d6ca
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
I20260923 14:36:32.510325 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:32.499614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:47083 (local address 127.2.218.62:37149)
0923 14:36:32.499749 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:32.499752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:32.500465 (+ 713us) server_negotiation.cc:407] Connection header received
0923 14:36:32.501726 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:32.501729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:32.501782 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:32.501886 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:32.503053 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.503526 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.504648 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.504866 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.504939 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:32.507325 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:32.507329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:32.507330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:32.507350 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:32.508563 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.508991 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.508995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.508997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.509069 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.509364 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.509368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.509370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.509524 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:32.509618 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:32.510046 (+ 428us) server_negotiation.cc:299] Negotiation successful
0923 14:36:32.510179 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 14:36:32.510900 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:32.499885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:37149 (local address 127.2.218.1:47083)
0923 14:36:32.500250 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:32.500284 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:32.501466 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:32.502047 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:32.502053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:32.502401 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:32.502921 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:32.502939 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.503640 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.503643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:32.504534 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:32.504546 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.505006 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.505012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:32.505199 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:32.505741 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:32.505755 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:32.507181 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:32.508669 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:32.508674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:32.508684 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:32.508898 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:32.509153 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:32.509156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:32.509157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:32.509280 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:32.509617 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:32.509623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:32.509816 (+ 193us) client_negotiation.cc:769] Sending connection context
0923 14:36:32.510004 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 14:36:32.510286 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
I20260923 14:36:32.511904 6701 heartbeater.cc:347] Connected to a master server at 127.2.218.62:37149
I20260923 14:36:32.512107 6701 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:32.512488 6701 heartbeater.cc:511] Master 127.2.218.62:37149 requested a full tablet report, sending...
I20260923 14:36:32.513872 6510 ts_manager.cc:194] Registered new tserver with Master: 3b653532852d44b3b9d7aedbaf31964a (127.2.218.1:37631)
I20260923 14:36:32.515097 6510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:47083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:36:32.524463 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37149
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:32.619032 6706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:32.619217 6706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:32.619251 6706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:32.622550 6706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:32.622591 6706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:32.622637 6706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:32.626701 6706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:32.627380 6706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:32.628088 6706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:32.635948 6714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:32.635947 6712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:32.635943 6711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:32.636188 6706 server_base.cc:1034] running on GCE node
I20260923 14:36:32.636543 6706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:32.637017 6706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:32.638160 6706 hybrid_clock.cc:648] HybridClock initialized: now 1790174192638139 us; error 35 us; skew 500 ppm
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:32.640436 6706 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:36:32.641145 6706 webserver.cc:460] Webserver started at http://127.2.218.2:46663/ using document root <none> and password file <none>
I20260923 14:36:32.641583 6706 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:32.641666 6706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:32.641851 6706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:32.643464 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "ae9c00730fc74de7a32cb6db2d25e8ea"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "2d8ee680fcbe0a83474748b3cdc1829c"
server_key_iv: "6175421058ee45c7b8982b1eb60664f9"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.643945 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "ae9c00730fc74de7a32cb6db2d25e8ea"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "2d8ee680fcbe0a83474748b3cdc1829c"
server_key_iv: "6175421058ee45c7b8982b1eb60664f9"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.646840 6706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:36:32.648792 6721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:32.649721 6706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:32.649839 6706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "ae9c00730fc74de7a32cb6db2d25e8ea"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "2d8ee680fcbe0a83474748b3cdc1829c"
server_key_iv: "6175421058ee45c7b8982b1eb60664f9"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.649932 6706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:32.692154 6706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:32.695021 6706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:32.695222 6706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:32.695739 6706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:32.696424 6706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:32.696491 6706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:32.696579 6706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:32.696612 6706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:32.704808 6706 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:39547
I20260923 14:36:32.704839 6834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:39547 every 8 connection(s)
I20260923 14:36:32.705489 6706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:36:32.709543 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 6706
I20260923 14:36:32.709659 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:32.709921 2920 external_mini_cluster.cc:1428] Setting key 07a4ccaad69420a96d6d6299e7eba8b6
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:32.717381 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:32.706945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:48253 (local address 127.2.218.62:37149)
0923 14:36:32.707063 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:32.707066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:32.707714 (+ 648us) server_negotiation.cc:407] Connection header received
0923 14:36:32.709036 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:32.709039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:32.709094 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:32.709199 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:32.710328 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.710882 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.711940 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.712276 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.712386 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:32.714385 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:32.714389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:32.714391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:32.714424 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:32.715632 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.716081 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.716088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.716089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.716163 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.716458 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.716461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.716462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.716619 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:32.716721 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:32.717128 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 14:36:32.717237 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
I20260923 14:36:32.718011 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:32.707233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:37149 (local address 127.2.218.2:48253)
0923 14:36:32.707585 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:32.707619 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:32.708836 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:32.709350 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:32.709356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:32.709626 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:32.710202 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:32.710215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.711033 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.711038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:32.711825 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:32.711833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.712397 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.712399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:32.712543 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:32.712990 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:32.713003 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:32.714241 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:32.715747 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:32.715752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:32.715763 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:32.715982 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:32.716247 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:32.716250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:32.716252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:32.716371 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:32.716720 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:32.716725 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:32.716930 (+ 205us) client_negotiation.cc:769] Sending connection context
0923 14:36:32.717084 (+ 154us) client_negotiation.cc:240] Negotiation successful
0923 14:36:32.717335 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
I20260923 14:36:32.718976 6835 heartbeater.cc:347] Connected to a master server at 127.2.218.62:37149
I20260923 14:36:32.719199 6835 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:32.719605 6835 heartbeater.cc:511] Master 127.2.218.62:37149 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
I20260923 14:36:32.720502 6510 ts_manager.cc:194] Registered new tserver with Master: ae9c00730fc74de7a32cb6db2d25e8ea (127.2.218.2:39547)
I20260923 14:36:32.721041 6510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:48253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:36:32.732544 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37149
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:32.826073 6840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:32.826314 6840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:32.826367 6840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:32.829691 6840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:32.829733 6840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:32.829813 6840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:32.834233 6840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:32.834954 6840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:32.835706 6840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:32.843247 6848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:32.843243 6846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:32.843235 6845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:32.843309 6840 server_base.cc:1034] running on GCE node
I20260923 14:36:32.843755 6840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:32.844237 6840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:32.845381 6840 hybrid_clock.cc:648] HybridClock initialized: now 1790174192845366 us; error 28 us; skew 500 ppm
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:32.847599 6840 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:36:32.848278 6840 webserver.cc:460] Webserver started at http://127.2.218.3:43181/ using document root <none> and password file <none>
I20260923 14:36:32.848708 6840 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:32.848783 6840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:32.848971 6840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:32.850572 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "b75a7a6e1688446fa51f8bd29f89f021"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "49957e3f2e66826ca9a28ec2547151ed"
server_key_iv: "6fa8889939d71c3d8bf773448b76e5bd"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.851016 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "b75a7a6e1688446fa51f8bd29f89f021"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "49957e3f2e66826ca9a28ec2547151ed"
server_key_iv: "6fa8889939d71c3d8bf773448b76e5bd"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.853977 6840 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:36:32.856022 6855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:32.857048 6840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:32.857158 6840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "b75a7a6e1688446fa51f8bd29f89f021"
format_stamp: "Formatted at 2026-09-23 14:36:32 on dist-test-slave-zl47"
server_key: "49957e3f2e66826ca9a28ec2547151ed"
server_key_iv: "6fa8889939d71c3d8bf773448b76e5bd"
server_key_version: "encryptionkey@0"
I20260923 14:36:32.857240 6840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:32.868953 6840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:32.871665 6840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:32.871822 6840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:32.872265 6840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:32.872982 6840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:32.873023 6840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:32.873072 6840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:32.873106 6840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:32.880448 6840 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:35091
I20260923 14:36:32.880513 6968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:35091 every 8 connection(s)
I20260923 14:36:32.881209 6840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:36:32.887002 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 6840
I20260923 14:36:32.887086 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:32.887312 2920 external_mini_cluster.cc:1428] Setting key 63bf5415044ca8468388a4e87e5b7bc7
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:32.892961 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:32.882889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:56291 (local address 127.2.218.62:37149)
0923 14:36:32.883034 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:32.883039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:32.883667 (+ 628us) server_negotiation.cc:407] Connection header received
0923 14:36:32.884710 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:32.884712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:32.884753 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:32.884817 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:32.885911 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.886408 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.887499 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.887726 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.887837 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:32.890024 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:32.890031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:32.890033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:32.890053 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:32.891261 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.891688 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.891692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.891693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.891766 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.892053 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.892057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.892058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.892232 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:32.892325 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:32.892719 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 14:36:32.892821 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 14:36:32.893520 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:32.883179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:37149 (local address 127.2.218.3:56291)
0923 14:36:32.883529 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:32.883561 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:32.884540 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:32.884942 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:32.884947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:32.885211 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:32.885787 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:32.885798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.886508 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.886510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:32.887381 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:32.887399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.887863 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.887867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:32.888008 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:32.888505 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:32.888522 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:32.889883 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:32.891364 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:32.891368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:32.891379 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:32.891596 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:32.891848 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:32.891851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:32.891853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:32.891969 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:32.892323 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:32.892329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:32.892521 (+ 192us) client_negotiation.cc:769] Sending connection context
0923 14:36:32.892682 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 14:36:32.892926 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":99,"threads_started":1}
I20260923 14:36:32.894477 6969 heartbeater.cc:347] Connected to a master server at 127.2.218.62:37149
I20260923 14:36:32.894631 6969 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:32.894953 6969 heartbeater.cc:511] Master 127.2.218.62:37149 requested a full tablet report, sending...
I20260923 14:36:32.895745 6510 ts_manager.cc:194] Registered new tserver with Master: b75a7a6e1688446fa51f8bd29f89f021 (127.2.218.3:35091)
I20260923 14:36:32.896194 6510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:56291
I20260923 14:36:32.900554 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:36:32.923070 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:32.915244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.2.218.1:37631)
0923 14:36:32.915542 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:32.915550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:32.915570 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:36:32.915609 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:32.915612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:32.915737 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:32.915830 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:32.916404 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.917025 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.917740 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.918013 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.918081 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:32.920095 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:32.920108 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:32.920116 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:32.920135 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:32.921588 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.921933 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.921936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.921937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.921999 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.922267 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.922269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.922270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.922443 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:32.922563 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:32.922744 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 14:36:32.922876 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":131,"threads_started":1}
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:32.931743 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:32.925356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.2.218.62:37149)
0923 14:36:32.925451 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:32.925454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:32.925485 (+ 31us) server_negotiation.cc:407] Connection header received
0923 14:36:32.925560 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:32.925562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:32.925589 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:32.925644 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:32.926191 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.926629 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.927262 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:32.927444 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:32.927511 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:32.929299 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:32.929301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:32.929303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:32.929321 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:32.930499 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.930821 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.930823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.930824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.930885 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:32.931075 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:32.931077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:32.931078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:32.931253 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:32.931337 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:32.931467 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 14:36:32.931557 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260923 14:36:32.932058 6510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37002
I20260923 14:36:32.932607 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 6572
I20260923 14:36:32.937459 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 6706
I20260923 14:36:32.941802 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 6840
I20260923 14:36:32.946574 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 6478
2026-09-23T14:36:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1841 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6981](info): set up 2 sockets
Sep 23 14:36:32 dist-test-slave-zl47 krb5kdc[6981](info): commencing operation
krb5kdc: starting...
W20260923 14:36:33.834043 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.867s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:33 dist-test-slave-zl47 krb5kdc[6981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:33.933178 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:44469
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:44469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:34.024899 6994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:34.025152 6994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:34.025214 6994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:34.028654 6994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:34.028728 6994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:34.028762 6994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:34.028805 6994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:34.028841 6994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:34.033034 6994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:44469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:44469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:34.033986 6994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:34.035377 6994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:34.044010 7000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:34.044019 7002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:34.044023 6999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:34.044392 6994 server_base.cc:1034] running on GCE node
I20260923 14:36:34.044850 6994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:34.045778 6994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:34.046954 6994 hybrid_clock.cc:648] HybridClock initialized: now 1790174194046940 us; error 29 us; skew 500 ppm
Sep 23 14:36:34 dist-test-slave-zl47 krb5kdc[6981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:34.049634 6994 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:34.050348 6994 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:34.050410 6994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:34.050545 6994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:34.052152 6994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "12921d040176c7ea1c0d3c67aaa94d8a"
server_key_iv: "9d6d0db8b6476ddb7ea289c0409721ab"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.052595 6994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "12921d040176c7ea1c0d3c67aaa94d8a"
server_key_iv: "9d6d0db8b6476ddb7ea289c0409721ab"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.055727 6994 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:34.057684 7008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:34.058763 6994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:34.058866 6994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "12921d040176c7ea1c0d3c67aaa94d8a"
server_key_iv: "9d6d0db8b6476ddb7ea289c0409721ab"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.058940 6994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:34.067884 6994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:34.070582 6994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:34.070715 6994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:34.076678 6994 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:44469
I20260923 14:36:34.076891 7060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:44469 every 8 connection(s)
I20260923 14:36:34.077244 6994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:34.079442 7061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:34.084076 7061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c: Bootstrap starting.
I20260923 14:36:34.086140 7061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:34.086776 7061 log.cc:824] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:34.087316 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 6994
I20260923 14:36:34.087435 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:34.087693 2920 external_mini_cluster.cc:1428] Setting key 38b8372e2b5cedc03627164d808367a0
I20260923 14:36:34.088559 7061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c: No bootstrap required, opened a new log
I20260923 14:36:34.092676 7061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44469 } }
I20260923 14:36:34.092845 7061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:34.092905 7061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b47ef6a178244f8b6bbf3a8d7a5338c, State: Initialized, Role: FOLLOWER
I20260923 14:36:34.093272 7061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44469 } }
I20260923 14:36:34.093403 7061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:34.093456 7061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:34.093528 7061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:34.094412 7061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44469 } }
I20260923 14:36:34.094755 7061 leader_election.cc:302] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b47ef6a178244f8b6bbf3a8d7a5338c; no voters:
I20260923 14:36:34.095057 7061 leader_election.cc:288] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:34.095199 7066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:34.095427 7066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [term 1 LEADER]: Becoming Leader. State: Replica: 1b47ef6a178244f8b6bbf3a8d7a5338c, State: Running, Role: LEADER
I20260923 14:36:34.095687 7066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44469 } }
Sep 23 14:36:34 dist-test-slave-zl47 krb5kdc[6981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:34.096174 7061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:34.096685 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b47ef6a178244f8b6bbf3a8d7a5338c. Latest consensus state: current_term: 1 leader_uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44469 } } }
I20260923 14:36:34.096792 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:34.097410 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b47ef6a178244f8b6bbf3a8d7a5338c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44469 } } }
I20260923 14:36:34.097520 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:34.098394 7071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:34.101404 7071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:34.101408 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.088668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.2.218.62:44469)
0923 14:36:34.089027 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:34.089043 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:34.089072 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:36:34.089616 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:34.089632 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:34.089910 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:34.090222 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:34.091006 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.092350 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.093293 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.093913 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.094054 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.096599 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:34.096605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:34.096618 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:34.096661 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:34.099017 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:34.099372 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:34.099377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:34.099382 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:34.099503 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:34.099774 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:34.099778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:34.099779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:34.100102 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:34.100316 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:34.100554 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 14:36:34.100801 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":113,"threads_started":1}
I20260923 14:36:34.105772 7071 catalog_manager.cc:1347] Generated new cluster ID: 7bb7061daee346a2853428852bb7f2a2
I20260923 14:36:34.105826 7071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:34.114811 7071 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:34.115734 7071 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:34.135818 7071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c: Generated new TSK 0
I20260923 14:36:34.136396 7071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:36:34.166999 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44469
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:34.289697 7087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:34.289880 7087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:34.289914 7087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:34.293256 7087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:34.293303 7087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:34.293350 7087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:34.297645 7087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:34.298484 7087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:34.299319 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:34.307418 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:34.307466 7087 server_base.cc:1034] running on GCE node
W20260923 14:36:34.307432 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:34.307435 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:34.307991 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:34.308502 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:34.309649 7087 hybrid_clock.cc:648] HybridClock initialized: now 1790174194309635 us; error 22 us; skew 500 ppm
Sep 23 14:36:34 dist-test-slave-zl47 krb5kdc[6981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:34.315189 7087 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:36:34.315757 7087 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:34.315810 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:34.315936 7087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:34.317454 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "8b7f2e63f60dff9eeabc2e1601799110"
server_key_iv: "ab6bda47d62be619d2b4b54034cf4724"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.317860 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "8b7f2e63f60dff9eeabc2e1601799110"
server_key_iv: "ab6bda47d62be619d2b4b54034cf4724"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.320840 7087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:34.322891 7101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:34.323854 7087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:34.323952 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "8b7f2e63f60dff9eeabc2e1601799110"
server_key_iv: "ab6bda47d62be619d2b4b54034cf4724"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.324033 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:34.334245 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:34.336376 7087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:34.336529 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:34.336992 7087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:34.337625 7087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:34.337663 7087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:34.337718 7087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:34.337743 7087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:34.345057 7087 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:43357
I20260923 14:36:34.345075 7214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:43357 every 8 connection(s)
I20260923 14:36:34.345490 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
Sep 23 14:36:34 dist-test-slave-zl47 krb5kdc[6981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:34.354753 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 7087
I20260923 14:36:34.354888 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:34.355206 2920 external_mini_cluster.cc:1428] Setting key a1550449dc27d5b4c096043c2b53bb3a
I20260923 14:36:34.356985 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.346882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:55317 (local address 127.2.218.62:44469)
0923 14:36:34.346995 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:34.346999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:34.347635 (+ 636us) server_negotiation.cc:407] Connection header received
0923 14:36:34.348705 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:34.348708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:34.348759 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:34.348825 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:34.349797 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.350411 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.351419 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.351630 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.351723 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.353553 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:34.353562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:34.353564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:34.353585 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:34.354916 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:34.355483 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:34.355486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:34.355487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:34.355582 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:34.355960 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:34.355964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:34.355965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:34.356191 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:34.356280 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:34.356797 (+ 517us) server_negotiation.cc:299] Negotiation successful
0923 14:36:34.356886 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":44}
I20260923 14:36:34.357908 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.347140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:44469 (local address 127.2.218.1:55317)
0923 14:36:34.347499 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:34.347531 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:34.348523 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:34.348954 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:34.348960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:34.349230 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:34.349649 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.349661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.350535 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.350551 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.351306 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.351314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.351761 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.351765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.351900 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.352271 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:34.352284 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:34.353415 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:34.355060 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:34.355067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:34.355079 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:34.355397 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:34.355688 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:34.355690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:34.355691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:34.355860 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:34.356280 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:34.356284 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:34.356563 (+ 279us) client_negotiation.cc:769] Sending connection context
0923 14:36:34.356757 (+ 194us) client_negotiation.cc:240] Negotiation successful
0923 14:36:34.357021 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260923 14:36:34.359019 7215 heartbeater.cc:347] Connected to a master server at 127.2.218.62:44469
I20260923 14:36:34.359223 7215 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:34.359561 7215 heartbeater.cc:511] Master 127.2.218.62:44469 requested a full tablet report, sending...
I20260923 14:36:34.360859 7025 ts_manager.cc:194] Registered new tserver with Master: 38bf652cd5bc48e0a8d7a7c7b726b26e (127.2.218.1:43357)
I20260923 14:36:34.361984 7025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:55317
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:36:34.379252 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44469
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:34.471310 7220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:34.471521 7220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:34.471587 7220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:34.475180 7220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:34.475224 7220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:34.475299 7220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:34.479707 7220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:34.480445 7220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:34.481230 7220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:34.489068 7226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:34.489089 7225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:34.489125 7220 server_base.cc:1034] running on GCE node
W20260923 14:36:34.489066 7228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:34.489611 7220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:34.490173 7220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:34.491358 7220 hybrid_clock.cc:648] HybridClock initialized: now 1790174194491343 us; error 28 us; skew 500 ppm
Sep 23 14:36:34 dist-test-slave-zl47 krb5kdc[6981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:34.493722 7220 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:36:34.494379 7220 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:34.494459 7220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:34.494676 7220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:34.496462 7220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "2feec27aad6d47a99059510497063a56"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "a120d5ca7317318ef17a353bb9e6cb4c"
server_key_iv: "541d0f5dd2b45f5015fa506a3eed079a"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.496941 7220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "2feec27aad6d47a99059510497063a56"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "a120d5ca7317318ef17a353bb9e6cb4c"
server_key_iv: "541d0f5dd2b45f5015fa506a3eed079a"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.500090 7220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 14:36:34.502089 7234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:34.503177 7220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:34.503309 7220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "2feec27aad6d47a99059510497063a56"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "a120d5ca7317318ef17a353bb9e6cb4c"
server_key_iv: "541d0f5dd2b45f5015fa506a3eed079a"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.503415 7220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:34.520352 7220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:34.522697 7220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:34.522856 7220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:34.523334 7220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:34.523981 7220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:34.524044 7220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:34.524099 7220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:34.524147 7220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:34.531860 7220 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:45111
I20260923 14:36:34.531883 7347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:45111 every 8 connection(s)
I20260923 14:36:34.532366 7220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:36:34.533507 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 7220
I20260923 14:36:34.533641 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:34.533912 2920 external_mini_cluster.cc:1428] Setting key 8b0affe0593d1ba4db501f1193cce166
Sep 23 14:36:34 dist-test-slave-zl47 krb5kdc[6981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
I20260923 14:36:34.543740 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.533814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:36619 (local address 127.2.218.62:44469)
0923 14:36:34.533925 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:34.533929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:34.534735 (+ 806us) server_negotiation.cc:407] Connection header received
0923 14:36:34.535959 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:34.535962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:34.536007 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:34.536088 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:34.536986 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.537462 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.538494 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.538698 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.538794 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.540772 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:34.540775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:34.540777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:34.540803 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:34.542042 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:34.542478 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:34.542483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:34.542484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:34.542553 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:34.542848 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:34.542852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:34.542854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:34.542999 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:34.543090 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:34.543491 (+ 401us) server_negotiation.cc:299] Negotiation successful
0923 14:36:34.543591 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:36:34.544317 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.534198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:44469 (local address 127.2.218.2:36619)
0923 14:36:34.534621 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:34.534654 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:34.535782 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:34.536237 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:34.536243 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:34.536510 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:34.536866 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.536876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.537572 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.537577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.538385 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.538393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.538821 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.538825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.539004 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.539486 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:34.539499 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:34.540638 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:34.542166 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:34.542171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:34.542181 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:34.542386 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:34.542649 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:34.542652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:34.542654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:34.542765 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:34.543089 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:34.543095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:34.543296 (+ 201us) client_negotiation.cc:769] Sending connection context
0923 14:36:34.543453 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 14:36:34.543691 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260923 14:36:34.545297 7348 heartbeater.cc:347] Connected to a master server at 127.2.218.62:44469
I20260923 14:36:34.545500 7348 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:34.545912 7348 heartbeater.cc:511] Master 127.2.218.62:44469 requested a full tablet report, sending...
I20260923 14:36:34.546811 7025 ts_manager.cc:194] Registered new tserver with Master: 2feec27aad6d47a99059510497063a56 (127.2.218.2:45111)
I20260923 14:36:34.547277 7025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:36619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:36:34.556236 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44469
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:34.648149 7353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:34.648326 7353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:34.648360 7353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:34.651700 7353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:34.651741 7353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:34.651787 7353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:34.655827 7353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:34.656471 7353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:34.657164 7353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:36:34.664722 7353 server_base.cc:1034] running on GCE node
W20260923 14:36:34.664681 7358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:34.664680 7359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:34.664690 7361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:34.665262 7353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:34.665707 7353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:34.666852 7353 hybrid_clock.cc:648] HybridClock initialized: now 1790174194666835 us; error 26 us; skew 500 ppm
Sep 23 14:36:34 dist-test-slave-zl47 krb5kdc[6981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:34.669230 7353 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:36:34.669903 7353 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:34.669992 7353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:34.670245 7353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:34.672008 7353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "e90fbd16bb3f47709d656b000b1c4b70"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "33b1fe0eb50123febe2a4e844e597964"
server_key_iv: "49e12ae3dfcd1cb3731143ffeddaec50"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.672444 7353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "e90fbd16bb3f47709d656b000b1c4b70"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "33b1fe0eb50123febe2a4e844e597964"
server_key_iv: "49e12ae3dfcd1cb3731143ffeddaec50"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.675316 7353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260923 14:36:34.677469 7367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:34.678498 7353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:34.678607 7353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "e90fbd16bb3f47709d656b000b1c4b70"
format_stamp: "Formatted at 2026-09-23 14:36:34 on dist-test-slave-zl47"
server_key: "33b1fe0eb50123febe2a4e844e597964"
server_key_iv: "49e12ae3dfcd1cb3731143ffeddaec50"
server_key_version: "encryptionkey@0"
I20260923 14:36:34.678716 7353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:34.688699 7353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:34.691375 7353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:34.691535 7353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:34.691962 7353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:34.692616 7353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:34.692678 7353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:34.692734 7353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:34.692778 7353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:34.700417 7353 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:39927
I20260923 14:36:34.700418 7480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:39927 every 8 connection(s)
I20260923 14:36:34.700932 7353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
Sep 23 14:36:34 dist-test-slave-zl47 krb5kdc[6981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:34.710667 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 7353
I20260923 14:36:34.710774 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:34.711004 2920 external_mini_cluster.cc:1428] Setting key 199bd4249f2b09d4940064ae6473534e
I20260923 14:36:34.712049 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.702413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:33875 (local address 127.2.218.62:44469)
0923 14:36:34.702532 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:34.702536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:34.703188 (+ 652us) server_negotiation.cc:407] Connection header received
0923 14:36:34.704233 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:34.704235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:34.704276 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:34.704339 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:34.705289 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.705736 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.706770 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.706957 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.707027 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.708954 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:34.708956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:34.708958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:34.708978 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:34.710258 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:34.710690 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:34.710694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:34.710696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:34.710783 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:34.711085 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:34.711088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:34.711089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:34.711304 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:34.711399 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:34.711804 (+ 405us) server_negotiation.cc:299] Negotiation successful
0923 14:36:34.711907 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:36:34.712702 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.702686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:44469 (local address 127.2.218.3:33875)
0923 14:36:34.703040 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:34.703072 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:34.704067 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:34.704516 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:34.704523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:34.704818 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:34.705175 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.705186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.705831 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.705833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.706676 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.706685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.707063 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.707065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.707177 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.707617 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:34.707647 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:34.708820 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:34.710364 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:34.710368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:34.710377 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:34.710591 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:34.710886 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:34.710888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:34.710889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:34.710992 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:34.711410 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:34.711416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:34.711603 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 14:36:34.711766 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 14:36:34.712005 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
I20260923 14:36:34.713505 7481 heartbeater.cc:347] Connected to a master server at 127.2.218.62:44469
I20260923 14:36:34.713690 7481 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:34.714026 7481 heartbeater.cc:511] Master 127.2.218.62:44469 requested a full tablet report, sending...
I20260923 14:36:34.714828 7025 ts_manager.cc:194] Registered new tserver with Master: e90fbd16bb3f47709d656b000b1c4b70 (127.2.218.3:39927)
I20260923 14:36:34.715301 7025 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:33875
I20260923 14:36:34.724090 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:36:34.731704 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.725734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.2.218.62:44469)
0923 14:36:34.725886 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:34.725891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:34.725974 (+ 83us) server_negotiation.cc:407] Connection header received
0923 14:36:34.726050 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:34.726052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:34.726082 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:34.726182 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:34.726775 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.727200 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.727909 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.728081 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.728148 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.729303 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:34.729305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:34.729307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:34.729321 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:34.730454 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:34.730786 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:34.730788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:34.730789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:34.730847 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:34.731039 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:34.731041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:34.731042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:34.731199 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:34.731285 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:34.731464 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 14:36:34.731535 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 14:36:34.734002 7025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:34.736661 7025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:34.748749 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.743668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:45111 (local address 127.0.0.1:56808)
0923 14:36:34.744309 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:34.744347 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:34.744477 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:34.745157 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:34.745165 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:34.745189 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:34.745337 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.745343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.746646 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.746649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.747510 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.747518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.748277 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.748280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.748380 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.748414 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 14:36:34.748523 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 14:36:34.748621 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":34,"spinlock_wait_cycles":1408,"thread_start_us":73,"threads_started":1}
I20260923 14:36:34.748754 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.743790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.2.218.2:45111)
0923 14:36:34.744130 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:34.744141 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:34.744430 (+ 289us) server_negotiation.cc:407] Connection header received
0923 14:36:34.744621 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:34.744626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:34.744811 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:34.744920 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:34.745454 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.746480 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.747631 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.748165 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.748274 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.748398 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:34.748548 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 14:36:34.748635 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260923 14:36:34.749142 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.744016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:39927 (local address 127.0.0.1:39884)
0923 14:36:34.744775 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:34.744785 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:34.744850 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:34.745396 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:34.745398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:34.745414 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:34.745497 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.745506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.746412 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.746417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.747674 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.747684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.748747 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.748750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.748871 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.748901 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 14:36:34.748962 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 14:36:34.749024 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":36,"thread_start_us":77,"threads_started":1}
I20260923 14:36:34.749328 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.743768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:43357 (local address 127.0.0.1:54968)
0923 14:36:34.744353 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:34.744364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:34.744460 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:34.745188 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:34.745190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:34.745199 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:34.745337 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.745343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.746603 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.746605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.747677 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:34.747684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.748747 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.748750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:34.748975 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.748991 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 14:36:34.749049 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 14:36:34.749097 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":117,"thread_start_us":77,"threads_started":1}
I20260923 14:36:34.749331 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.744312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.2.218.1:43357)
0923 14:36:34.744669 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:34.744682 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:34.744705 (+ 23us) server_negotiation.cc:407] Connection header received
0923 14:36:34.744774 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:34.744778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:34.744943 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:34.745077 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:34.745456 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.746478 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.747805 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.748484 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.748575 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.748686 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:34.749104 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 14:36:34.749205 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":89,"threads_started":1}
I20260923 14:36:34.749660 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:34.744333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.2.218.3:39927)
0923 14:36:34.745024 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:34.745036 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:34.745054 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:36:34.745111 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:34.745115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:34.745270 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:34.745411 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:34.745614 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.746300 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.747804 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:34.748484 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:34.748575 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:34.748685 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:34.749401 (+ 716us) server_negotiation.cc:299] Negotiation successful
0923 14:36:34.749495 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":71,"threads_started":1}
I20260923 14:36:34.751526 7282 tablet_service.cc:1467] Processing CreateTablet for tablet ec20558a4db843bbb22e8f61543eba33 (DEFAULT_TABLE table=test-table [id=9baf317a2cc54fc4b1b513e0feeb3d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:34.751729 7149 tablet_service.cc:1467] Processing CreateTablet for tablet ec20558a4db843bbb22e8f61543eba33 (DEFAULT_TABLE table=test-table [id=9baf317a2cc54fc4b1b513e0feeb3d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:34.751892 7415 tablet_service.cc:1467] Processing CreateTablet for tablet ec20558a4db843bbb22e8f61543eba33 (DEFAULT_TABLE table=test-table [id=9baf317a2cc54fc4b1b513e0feeb3d84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:34.752499 7149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec20558a4db843bbb22e8f61543eba33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:34.752499 7282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec20558a4db843bbb22e8f61543eba33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:34.752722 7415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec20558a4db843bbb22e8f61543eba33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:34.757666 7498 tablet_bootstrap.cc:492] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: Bootstrap starting.
I20260923 14:36:34.758939 7499 tablet_bootstrap.cc:492] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: Bootstrap starting.
I20260923 14:36:34.760028 7498 tablet_bootstrap.cc:654] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:34.760442 7499 tablet_bootstrap.cc:654] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:34.760993 7498 log.cc:824] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:34.761507 7499 log.cc:824] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:34.761521 7500 tablet_bootstrap.cc:492] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e: Bootstrap starting.
I20260923 14:36:34.762794 7498 tablet_bootstrap.cc:492] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: No bootstrap required, opened a new log
I20260923 14:36:34.762951 7498 ts_tablet_manager.cc:1397] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:36:34.763240 7499 tablet_bootstrap.cc:492] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: No bootstrap required, opened a new log
I20260923 14:36:34.763388 7499 ts_tablet_manager.cc:1397] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:36:34.763660 7500 tablet_bootstrap.cc:654] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:34.764544 7500 log.cc:824] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:34.765874 7500 tablet_bootstrap.cc:492] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e: No bootstrap required, opened a new log
I20260923 14:36:34.766063 7500 ts_tablet_manager.cc:1397] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:36:34.766330 7498 raft_consensus.cc:348] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 } } peers { permanent_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43357 } } peers { permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 } }
I20260923 14:36:34.766564 7498 raft_consensus.cc:374] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:34.766615 7498 raft_consensus.cc:729] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90fbd16bb3f47709d656b000b1c4b70, State: Initialized, Role: FOLLOWER
I20260923 14:36:34.766607 7499 raft_consensus.cc:348] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 } } peers { permanent_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43357 } } peers { permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 } }
I20260923 14:36:34.766824 7499 raft_consensus.cc:374] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:34.766863 7499 raft_consensus.cc:729] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2feec27aad6d47a99059510497063a56, State: Initialized, Role: FOLLOWER
I20260923 14:36:34.766996 7498 consensus_queue.cc:260] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 } } peers { permanent_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43357 } } peers { permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 } }
I20260923 14:36:34.767184 7499 consensus_queue.cc:260] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 } } peers { permanent_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43357 } } peers { permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 } }
I20260923 14:36:34.767565 7481 heartbeater.cc:503] Master 127.2.218.62:44469 was elected leader, sending a full tablet report...
I20260923 14:36:34.767930 7499 ts_tablet_manager.cc:1428] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260923 14:36:34.768187 7348 heartbeater.cc:503] Master 127.2.218.62:44469 was elected leader, sending a full tablet report...
I20260923 14:36:34.768879 7498 ts_tablet_manager.cc:1428] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:36:34.770722 7500 raft_consensus.cc:348] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 } } peers { permanent_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43357 } } peers { permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 } }
I20260923 14:36:34.770939 7500 raft_consensus.cc:374] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:34.770992 7500 raft_consensus.cc:729] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38bf652cd5bc48e0a8d7a7c7b726b26e, State: Initialized, Role: FOLLOWER
I20260923 14:36:34.771414 7500 consensus_queue.cc:260] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 } } peers { permanent_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43357 } } peers { permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 } }
I20260923 14:36:34.771935 7215 heartbeater.cc:503] Master 127.2.218.62:44469 was elected leader, sending a full tablet report...
I20260923 14:36:34.772181 7500 ts_tablet_manager.cc:1428] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
W20260923 14:36:34.783094 7349 tablet.cc:2369] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:36:34.846413 7216 tablet.cc:2369] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:36:34.951752 7482 tablet.cc:2369] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:35.106308 7506 raft_consensus.cc:482] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:35.106535 7506 raft_consensus.cc:504] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 } } peers { permanent_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43357 } } peers { permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 } }
I20260923 14:36:35.107616 7506 leader_election.cc:288] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2feec27aad6d47a99059510497063a56 (127.2.218.2:45111), e90fbd16bb3f47709d656b000b1c4b70 (127.2.218.3:39927)
I20260923 14:36:35.111348 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.107895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:53455 (local address 127.2.218.2:45111)
0923 14:36:35.108020 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.108024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.108225 (+ 201us) server_negotiation.cc:407] Connection header received
0923 14:36:35.108320 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.108322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.108366 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.108434 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:35.108738 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:35.109241 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:35.110376 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:35.110721 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:35.110837 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:35.110910 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.111145 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.111223 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:36:35.111375 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.107957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:43793 (local address 127.2.218.3:39927)
0923 14:36:35.108064 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.108068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.108365 (+ 297us) server_negotiation.cc:407] Connection header received
0923 14:36:35.108452 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.108455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.108506 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.108590 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:35.108836 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:35.109316 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:35.110333 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:35.110724 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:35.110859 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:35.110932 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.111154 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.111233 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:36:35.111404 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.107806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:39927 (local address 127.2.218.1:43793)
0923 14:36:35.108256 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:35.108267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:35.108353 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:35.108596 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:35.108598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:35.108610 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:35.108724 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:35.108729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:35.109418 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:35.109420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:35.110216 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:35.110224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:35.110870 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:35.110886 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:35.111013 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:35.111029 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 14:36:35.111118 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 14:36:35.111237 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":95,"thread_start_us":59,"threads_started":1}
I20260923 14:36:35.111430 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.107802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:45111 (local address 127.2.218.1:53455)
0923 14:36:35.108138 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:35.108150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:35.108238 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:35.108464 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:35.108466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:35.108481 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:35.108629 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:35.108635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:35.109351 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:35.109357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:35.110242 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:35.110251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:35.110853 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:35.110857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:35.110978 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:35.111010 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 14:36:35.111122 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 14:36:35.111257 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":106,"thread_start_us":85,"threads_started":1}
I20260923 14:36:35.112020 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec20558a4db843bbb22e8f61543eba33" candidate_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2feec27aad6d47a99059510497063a56" is_pre_election: true
I20260923 14:36:35.112080 7435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec20558a4db843bbb22e8f61543eba33" candidate_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90fbd16bb3f47709d656b000b1c4b70" is_pre_election: true
I20260923 14:36:35.112317 7435 raft_consensus.cc:2454] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38bf652cd5bc48e0a8d7a7c7b726b26e in term 0.
I20260923 14:36:35.112317 7302 raft_consensus.cc:2454] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38bf652cd5bc48e0a8d7a7c7b726b26e in term 0.
I20260923 14:36:35.112771 7102 leader_election.cc:302] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2feec27aad6d47a99059510497063a56, 38bf652cd5bc48e0a8d7a7c7b726b26e, e90fbd16bb3f47709d656b000b1c4b70; no voters:
I20260923 14:36:35.113029 7506 raft_consensus.cc:2781] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:35.113096 7506 raft_consensus.cc:482] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:35.113122 7506 raft_consensus.cc:3037] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:35.114024 7506 raft_consensus.cc:504] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 } } peers { permanent_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43357 } } peers { permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 } }
I20260923 14:36:35.114353 7506 leader_election.cc:288] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [CANDIDATE]: Term 1 election: Requested vote from peers 2feec27aad6d47a99059510497063a56 (127.2.218.2:45111), e90fbd16bb3f47709d656b000b1c4b70 (127.2.218.3:39927)
I20260923 14:36:35.114648 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec20558a4db843bbb22e8f61543eba33" candidate_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2feec27aad6d47a99059510497063a56"
I20260923 14:36:35.114782 7435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec20558a4db843bbb22e8f61543eba33" candidate_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e90fbd16bb3f47709d656b000b1c4b70"
I20260923 14:36:35.114785 7302 raft_consensus.cc:3037] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:35.114894 7435 raft_consensus.cc:3037] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:35.115990 7435 raft_consensus.cc:2454] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38bf652cd5bc48e0a8d7a7c7b726b26e in term 1.
I20260923 14:36:35.115989 7302 raft_consensus.cc:2454] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38bf652cd5bc48e0a8d7a7c7b726b26e in term 1.
I20260923 14:36:35.116266 7102 leader_election.cc:302] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2feec27aad6d47a99059510497063a56, 38bf652cd5bc48e0a8d7a7c7b726b26e; no voters:
I20260923 14:36:35.116438 7506 raft_consensus.cc:2781] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:35.116636 7506 raft_consensus.cc:686] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [term 1 LEADER]: Becoming Leader. State: Replica: 38bf652cd5bc48e0a8d7a7c7b726b26e, State: Running, Role: LEADER
I20260923 14:36:35.116907 7506 consensus_queue.cc:237] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 } } peers { permanent_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43357 } } peers { permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 } }
I20260923 14:36:35.119935 7025 catalog_manager.cc:5495] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e reported cstate change: term changed from 0 to 1, leader changed from <none> to 38bf652cd5bc48e0a8d7a7c7b726b26e (127.2.218.1). New cstate: current_term: 1 leader_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38bf652cd5bc48e0a8d7a7c7b726b26e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 } health_report { overall_health: UNKNOWN } } }
I20260923 14:36:35.196548 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.193506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.2.218.1:43357)
0923 14:36:35.193630 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.193633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.193645 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:35.193732 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.193734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.193783 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.193859 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:35.194073 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:35.194546 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:35.195250 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:35.195420 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:35.195484 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:35.195777 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:35.196155 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:35.196235 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.196325 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.196372 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:36:35.201967 7435 raft_consensus.cc:1261] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70 [term 1 FOLLOWER]: Refusing update from remote peer 38bf652cd5bc48e0a8d7a7c7b726b26e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:35.201973 7302 raft_consensus.cc:1261] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56 [term 1 FOLLOWER]: Refusing update from remote peer 38bf652cd5bc48e0a8d7a7c7b726b26e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:35.202646 7509 consensus_queue.cc:1035] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e90fbd16bb3f47709d656b000b1c4b70" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:35.202819 7506 consensus_queue.cc:1035] T ec20558a4db843bbb22e8f61543eba33 P 38bf652cd5bc48e0a8d7a7c7b726b26e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2feec27aad6d47a99059510497063a56" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:35.217597 7025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:35.217718 7025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:35.219738 7025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b47ef6a178244f8b6bbf3a8d7a5338c: Sending DeleteTablet for 3 replicas of tablet ec20558a4db843bbb22e8f61543eba33
I20260923 14:36:35.220368 7282 tablet_service.cc:1514] Processing DeleteTablet for tablet ec20558a4db843bbb22e8f61543eba33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:35 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:56808
I20260923 14:36:35.220616 7149 tablet_service.cc:1514] Processing DeleteTablet for tablet ec20558a4db843bbb22e8f61543eba33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:35 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:54968
I20260923 14:36:35.220752 7415 tablet_service.cc:1514] Processing DeleteTablet for tablet ec20558a4db843bbb22e8f61543eba33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:35 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:39884
I20260923 14:36:35.221015 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 7087
I20260923 14:36:35.221455 7522 tablet_replica.cc:330] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: stopping tablet replica
I20260923 14:36:35.221716 7522 raft_consensus.cc:2229] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:35.221964 7522 raft_consensus.cc:2258] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:35.222642 7524 tablet_replica.cc:330] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: stopping tablet replica
I20260923 14:36:35.222915 7524 raft_consensus.cc:2229] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:35.223197 7524 raft_consensus.cc:2258] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:35.223737 7522 ts_tablet_manager.cc:1905] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:35.225600 7522 ts_tablet_manager.cc:1918] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:35.225705 7522 log.cc:1196] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/ec20558a4db843bbb22e8f61543eba33
I20260923 14:36:35.226052 7522 ts_tablet_manager.cc:1939] T ec20558a4db843bbb22e8f61543eba33 P 2feec27aad6d47a99059510497063a56: Deleting consensus metadata
I20260923 14:36:35.226747 7524 ts_tablet_manager.cc:1905] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 14:36:35.227203 7010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:43357 (error 108)
I20260923 14:36:35.227308 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 7220
I20260923 14:36:35.228575 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.228272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:43357 (local address 127.0.0.1:54974)
0923 14:36:35.228435 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:35.228504 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:43357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260923 14:36:35.228718 7010 catalog_manager.cc:4584] TS 38bf652cd5bc48e0a8d7a7c7b726b26e (127.2.218.1:43357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec20558a4db843bbb22e8f61543eba33: Network error: Client connection negotiation failed: client connection to 127.2.218.1:43357: connect: Connection refused (error 111)
I20260923 14:36:35.229283 7524 ts_tablet_manager.cc:1918] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:35.229378 7524 log.cc:1196] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/ec20558a4db843bbb22e8f61543eba33
I20260923 14:36:35.229610 7524 ts_tablet_manager.cc:1939] T ec20558a4db843bbb22e8f61543eba33 P e90fbd16bb3f47709d656b000b1c4b70: Deleting consensus metadata
I20260923 14:36:35.230508 7011 catalog_manager.cc:4841] TS e90fbd16bb3f47709d656b000b1c4b70 (127.2.218.3:39927): tablet ec20558a4db843bbb22e8f61543eba33 (table test-table [id=9baf317a2cc54fc4b1b513e0feeb3d84]) successfully deleted
I20260923 14:36:35.233006 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 7353
I20260923 14:36:35.234889 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.233174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:45111 (local address 127.0.0.1:56824)
0923 14:36:35.234710 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:35.234786 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.2:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1445}
I20260923 14:36:35.238147 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 6994
2026-09-23T14:36:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2291 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T14:36:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:35Z Disabled control of system clock
I20260923 14:36:35.265476 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:44385
--webserver_interface=127.2.218.62
--webserver_port=0
--builtin_ntp_servers=127.2.218.20:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:44385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 14:36:35.362208 7529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:35.362432 7529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:35.362463 7529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:35.365725 7529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:35.365763 7529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:35.365777 7529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:35.365790 7529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:35.365803 7529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:35.369765 7529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:44385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:35.370569 7529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:35.371671 7529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:35.379115 7537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:35.379156 7535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:35.379330 7534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:35.379413 7529 server_base.cc:1034] running on GCE node
I20260923 14:36:35.379957 7529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:35.380918 7529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:35.382067 7529 hybrid_clock.cc:648] HybridClock initialized: now 1790174195382054 us; error 23 us; skew 500 ppm
I20260923 14:36:35.383765 7529 webserver.cc:460] Webserver started at http://127.2.218.62:39733/ using document root <none> and password file <none>
I20260923 14:36:35.384415 7529 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:35.384478 7529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:35.384794 7529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:35.386653 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "ce49aa068500408291c9e190fde17837"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "71d1f1eb962192ed0688be924736bb7b"
server_key_iv: "c3959156ea9a771ecd4ce893e7135ca6"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.387168 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "ce49aa068500408291c9e190fde17837"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "71d1f1eb962192ed0688be924736bb7b"
server_key_iv: "c3959156ea9a771ecd4ce893e7135ca6"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.390296 7529 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:36:35.392282 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:35.393312 7529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:35.393435 7529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "ce49aa068500408291c9e190fde17837"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "71d1f1eb962192ed0688be924736bb7b"
server_key_iv: "c3959156ea9a771ecd4ce893e7135ca6"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.393532 7529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:35.397564 7529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:35.398173 7529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:35.398346 7529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:35.404196 7529 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:44385
I20260923 14:36:35.404199 7595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:44385 every 8 connection(s)
I20260923 14:36:35.404932 7529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:35.407655 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:35.409693 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 7529
I20260923 14:36:35.409812 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:35.410053 2920 external_mini_cluster.cc:1428] Setting key 5bfbdbc1bc0bb8c72ca294b86d1c9151
I20260923 14:36:35.412424 7596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837: Bootstrap starting.
I20260923 14:36:35.414559 7596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:35.414855 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.411384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.2.218.62:44385)
0923 14:36:35.411913 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.411926 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.411956 (+ 30us) server_negotiation.cc:407] Connection header received
0923 14:36:35.412462 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.412475 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.412805 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.413036 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:35.413742 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:35.413746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:35.413755 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:35.413763 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:35.413811 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:35.413824 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:35.413914 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.414008 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.414273 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":121,"threads_started":1}
I20260923 14:36:35.415310 7596 log.cc:824] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:35.417017 7596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837: No bootstrap required, opened a new log
I20260923 14:36:35.420009 7596 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49aa068500408291c9e190fde17837" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44385 } }
I20260923 14:36:35.420186 7596 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:35.420217 7596 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce49aa068500408291c9e190fde17837, State: Initialized, Role: FOLLOWER
I20260923 14:36:35.420523 7596 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49aa068500408291c9e190fde17837" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44385 } }
I20260923 14:36:35.420620 7596 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:35.420657 7596 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:35.420714 7596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:35.421397 7596 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49aa068500408291c9e190fde17837" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44385 } }
I20260923 14:36:35.421693 7596 leader_election.cc:302] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce49aa068500408291c9e190fde17837; no voters:
I20260923 14:36:35.421927 7596 leader_election.cc:288] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:35.422024 7601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:35.422325 7601 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [term 1 LEADER]: Becoming Leader. State: Replica: ce49aa068500408291c9e190fde17837, State: Running, Role: LEADER
I20260923 14:36:35.422618 7601 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49aa068500408291c9e190fde17837" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44385 } }
I20260923 14:36:35.423036 7596 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:35.424258 7602 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce49aa068500408291c9e190fde17837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49aa068500408291c9e190fde17837" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44385 } } }
I20260923 14:36:35.424383 7602 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:35.424691 7603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce49aa068500408291c9e190fde17837. Latest consensus state: current_term: 1 leader_uuid: "ce49aa068500408291c9e190fde17837" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce49aa068500408291c9e190fde17837" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44385 } } }
I20260923 14:36:35.424813 7603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:35.424753 7610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:35.427755 7610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:35.432962 7610 catalog_manager.cc:1347] Generated new cluster ID: 97a5dd474bfc4b2db36e928f8a4fae31
I20260923 14:36:35.433020 7610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:35.443203 7610 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:35.443470 7610 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:35.448295 7610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837: Generated new TSK 0
I20260923 14:36:35.448823 7610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:36:35.454377 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44385
--builtin_ntp_servers=127.2.218.20:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 14:36:35.549067 7620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:35.549252 7620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:35.549283 7620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:35.552517 7620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:35.552553 7620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:35.552597 7620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:35.556608 7620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:35.557314 7620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:35.558334 7620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:35.566041 7625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:35.566040 7626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:35.566040 7628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:35.566169 7620 server_base.cc:1034] running on GCE node
I20260923 14:36:35.566643 7620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:35.567082 7620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:35.568223 7620 hybrid_clock.cc:648] HybridClock initialized: now 1790174195568208 us; error 25 us; skew 500 ppm
I20260923 14:36:35.569754 7620 webserver.cc:460] Webserver started at http://127.2.218.1:38381/ using document root <none> and password file <none>
I20260923 14:36:35.570281 7620 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:35.570335 7620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:35.570540 7620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:35.572145 7620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "c0c718b0b791493494b783aa793504b7"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "b551bddd18ec444d13a85599f716afc5"
server_key_iv: "b2435160728d4eedfe607a2daff48cc5"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.572613 7620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "c0c718b0b791493494b783aa793504b7"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "b551bddd18ec444d13a85599f716afc5"
server_key_iv: "b2435160728d4eedfe607a2daff48cc5"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.575795 7620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 14:36:35.577663 7634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:35.578719 7620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:35.578837 7620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "c0c718b0b791493494b783aa793504b7"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "b551bddd18ec444d13a85599f716afc5"
server_key_iv: "b2435160728d4eedfe607a2daff48cc5"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.578931 7620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:35.582599 7620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:35.583069 7620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:35.583225 7620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:35.583659 7620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:35.584380 7620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:35.584441 7620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:35.584498 7620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:35.584542 7620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:35.591756 7620 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:42375
I20260923 14:36:35.591776 7747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:42375 every 8 connection(s)
I20260923 14:36:35.592417 7620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:35.597435 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.593848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:56085 (local address 127.2.218.62:44385)
0923 14:36:35.593961 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.593965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.594815 (+ 850us) server_negotiation.cc:407] Connection header received
0923 14:36:35.595871 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.595874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.595918 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.596008 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:35.597012 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:35.597015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:35.597017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:35.597027 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:35.597046 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:35.597053 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:35.597149 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.597279 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.597322 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:36:35.598243 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.594081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:44385 (local address 127.2.218.1:56085)
0923 14:36:35.594687 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:35.594718 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:35.595687 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:35.596205 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:35.596210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:35.596283 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:35.596807 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:36:35.596808 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:35.596838 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:36:35.596891 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:35.597142 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:35.597148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:35.597152 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:36:35.597307 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 14:36:35.597490 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20260923 14:36:35.599263 7748 heartbeater.cc:347] Connected to a master server at 127.2.218.62:44385
I20260923 14:36:35.599448 7748 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:35.599433 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 7620
I20260923 14:36:35.599567 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:35.599795 2920 external_mini_cluster.cc:1428] Setting key 9f7b97f732c66e6739827fb3dd3c85ef
I20260923 14:36:35.599823 7748 heartbeater.cc:511] Master 127.2.218.62:44385 requested a full tablet report, sending...
I20260923 14:36:35.601292 7559 ts_manager.cc:194] Registered new tserver with Master: c0c718b0b791493494b783aa793504b7 (127.2.218.1:42375)
I20260923 14:36:35.601610 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44385
--builtin_ntp_servers=127.2.218.20:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 14:36:35.693332 7751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:35.693521 7751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:35.693552 7751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:35.696983 7751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:35.697043 7751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:35.697090 7751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:35.701277 7751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:35.702013 7751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:35.703140 7751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:35.711079 7759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:35.711094 7756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:35.711156 7757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:35.711299 7751 server_base.cc:1034] running on GCE node
I20260923 14:36:35.711661 7751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:35.712142 7751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:35.713336 7751 hybrid_clock.cc:648] HybridClock initialized: now 1790174195713319 us; error 32 us; skew 500 ppm
I20260923 14:36:35.714977 7751 webserver.cc:460] Webserver started at http://127.2.218.2:40037/ using document root <none> and password file <none>
I20260923 14:36:35.715487 7751 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:35.715595 7751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:35.715817 7751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:35.717562 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "517c428d5da4412ca0dfa3bea4cf5584"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "216bbd3a6fd4b3b61f6e77aaabbfad5c"
server_key_iv: "282d4888f9048bc7795df59ed253ef56"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.718063 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "517c428d5da4412ca0dfa3bea4cf5584"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "216bbd3a6fd4b3b61f6e77aaabbfad5c"
server_key_iv: "282d4888f9048bc7795df59ed253ef56"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.721141 7751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:36:35.723223 7765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:35.724318 7751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:35.724434 7751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "517c428d5da4412ca0dfa3bea4cf5584"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "216bbd3a6fd4b3b61f6e77aaabbfad5c"
server_key_iv: "282d4888f9048bc7795df59ed253ef56"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.724524 7751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:35.728188 7751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:35.728649 7751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:35.728819 7751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:35.729274 7751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:35.730064 7751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:35.730151 7751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:35.730226 7751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:35.730273 7751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:35.737782 7751 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:37537
I20260923 14:36:35.737808 7878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:37537 every 8 connection(s)
I20260923 14:36:35.738566 7751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:36:35.743280 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.739945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:59493 (local address 127.2.218.62:44385)
0923 14:36:35.740062 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.740066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.740845 (+ 779us) server_negotiation.cc:407] Connection header received
0923 14:36:35.741841 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.741843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.741877 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.741941 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:35.742933 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:35.742937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:35.742939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:35.742950 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:35.742967 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:35.742974 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:35.743046 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.743152 (+ 106us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.743187 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:36:35.743968 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.740190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:44385 (local address 127.2.218.2:59493)
0923 14:36:35.740709 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:35.740740 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:35.741686 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:35.742067 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:35.742073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:35.742207 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:35.742693 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:36:35.742694 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:35.742747 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:36:35.742801 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:35.743045 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:35.743050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:35.743053 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:36:35.743175 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 14:36:35.743349 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260923 14:36:35.744940 7879 heartbeater.cc:347] Connected to a master server at 127.2.218.62:44385
I20260923 14:36:35.745134 7879 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:35.745440 7879 heartbeater.cc:511] Master 127.2.218.62:44385 requested a full tablet report, sending...
I20260923 14:36:35.746371 7559 ts_manager.cc:194] Registered new tserver with Master: 517c428d5da4412ca0dfa3bea4cf5584 (127.2.218.2:37537)
I20260923 14:36:35.746810 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 7751
I20260923 14:36:35.746925 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:35.747184 2920 external_mini_cluster.cc:1428] Setting key 0b41971045fe999c35445d8081958776
I20260923 14:36:35.748895 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44385
--builtin_ntp_servers=127.2.218.20:45689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 14:36:35.845927 7882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:35.846156 7882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:35.846231 7882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:35.849720 7882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:35.849766 7882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:35.849848 7882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:35.853835 7882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:35.854722 7882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:35.855882 7882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:35.864018 7888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:35.864022 7887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:35.864176 7882 server_base.cc:1034] running on GCE node
W20260923 14:36:35.864395 7890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:35.864734 7882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:35.865300 7882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:35.866447 7882 hybrid_clock.cc:648] HybridClock initialized: now 1790174195866419 us; error 40 us; skew 500 ppm
I20260923 14:36:35.868119 7882 webserver.cc:460] Webserver started at http://127.2.218.3:43963/ using document root <none> and password file <none>
I20260923 14:36:35.868572 7882 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:35.868628 7882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:35.868775 7882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:35.870544 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "beea9d5ce054403c9ea23df31b130181"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "76193e2482b6336cc6cb722751254865"
server_key_iv: "81f73a5794b7c5078642bdbdfb5b6c48"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.871006 7882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "beea9d5ce054403c9ea23df31b130181"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "76193e2482b6336cc6cb722751254865"
server_key_iv: "81f73a5794b7c5078642bdbdfb5b6c48"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.874024 7882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:36:35.876077 7896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:35.877089 7882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:35.877172 7882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "beea9d5ce054403c9ea23df31b130181"
format_stamp: "Formatted at 2026-09-23 14:36:35 on dist-test-slave-zl47"
server_key: "76193e2482b6336cc6cb722751254865"
server_key_iv: "81f73a5794b7c5078642bdbdfb5b6c48"
server_key_version: "encryptionkey@0"
I20260923 14:36:35.877243 7882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:35.881245 7882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:35.881707 7882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:35.881866 7882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:35.882407 7882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:35.883174 7882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:35.883221 7882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:35.883260 7882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:35.883283 7882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:35.891476 7882 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:43015
I20260923 14:36:35.891500 8009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:43015 every 8 connection(s)
I20260923 14:36:35.892277 7882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:36:35.893705 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 7882
I20260923 14:36:35.893816 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:35.894043 2920 external_mini_cluster.cc:1428] Setting key 5c33140ea89c1946ece1580d7b0f624f
I20260923 14:36:35.897404 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.893846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:48501 (local address 127.2.218.62:44385)
0923 14:36:35.893968 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.893973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.894989 (+ 1016us) server_negotiation.cc:407] Connection header received
0923 14:36:35.896036 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.896038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.896071 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.896130 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:35.897086 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:35.897088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:35.897090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:35.897099 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:35.897113 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:35.897120 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:35.897181 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.897280 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.897319 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:36:35.898065 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.894261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:44385 (local address 127.2.218.3:48501)
0923 14:36:35.894852 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:35.894894 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:35.895886 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:35.896235 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:35.896241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:35.896311 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:35.896863 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:36:35.896864 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:35.896897 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:36:35.896967 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:35.897177 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:35.897182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:35.897185 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:36:35.897299 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 14:36:35.897469 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":119,"threads_started":1}
I20260923 14:36:35.898978 8010 heartbeater.cc:347] Connected to a master server at 127.2.218.62:44385
I20260923 14:36:35.899139 8010 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:35.899402 8010 heartbeater.cc:511] Master 127.2.218.62:44385 requested a full tablet report, sending...
I20260923 14:36:35.900193 7559 ts_manager.cc:194] Registered new tserver with Master: beea9d5ce054403c9ea23df31b130181 (127.2.218.3:43015)
I20260923 14:36:35.907239 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:36:35.910334 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.909034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.2.218.62:44385)
0923 14:36:35.909145 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.909148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.909312 (+ 164us) server_negotiation.cc:407] Connection header received
0923 14:36:35.909390 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.909392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.909414 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.909469 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:35.909999 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:35.910001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:35.910003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:35.910012 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:35.910028 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:35.910034 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:35.910176 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.910228 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.910256 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:36:35.912518 7559 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:35.914995 7559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:35.924017 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.921194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:43015 (local address 127.0.0.1:40110)
0923 14:36:35.921502 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:35.921541 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:35.921603 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:35.922411 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:35.922417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:35.922480 (+ 63us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:35.922947 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:36:35.922951 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:35.922969 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:36:35.923028 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:35.923746 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:35.923754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:35.923758 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:36:35.923858 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 14:36:35.923916 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260923 14:36:35.924345 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.921309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.2.218.3:43015)
0923 14:36:35.921594 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.921614 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.921633 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:36:35.921671 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.921675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.921819 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.921934 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:35.923593 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:35.923610 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:35.923619 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:35.923628 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:35.923660 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:35.923681 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:35.923772 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.923940 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.923993 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":66,"threads_started":1}
I20260923 14:36:35.924408 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.922048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:42375 (local address 127.0.0.1:38516)
0923 14:36:35.922320 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:35.922332 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:35.922400 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:35.923075 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:35.923078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:35.923090 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:35.923456 (+ 366us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:36:35.923458 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:35.923467 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:36:35.923490 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:35.924068 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:35.924071 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:35.924235 (+ 164us) client_negotiation.cc:769] Sending connection context
0923 14:36:35.924290 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 14:36:35.924314 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":87,"threads_started":1}
I20260923 14:36:35.924680 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.921811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:37537 (local address 127.0.0.1:57366)
0923 14:36:35.922288 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:35.922301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:35.922369 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:35.923718 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:35.923722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:35.923739 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:35.924113 (+ 374us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:36:35.924115 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:35.924132 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:36:35.924157 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:35.924480 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:35.924483 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:35.924486 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:36:35.924555 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 14:36:35.924585 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":60,"threads_started":1}
I20260923 14:36:35.924835 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.921942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.2.218.2:37537)
0923 14:36:35.922710 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.922721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.922742 (+ 21us) server_negotiation.cc:407] Connection header received
0923 14:36:35.922789 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.922795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.922949 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.923095 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:35.924270 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:35.924276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:35.924284 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:35.924294 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:35.924326 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:35.924353 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:35.924443 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.924579 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.924641 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":104,"threads_started":1}
I20260923 14:36:35.924903 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:35.922231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.2.218.1:42375)
0923 14:36:35.922640 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:35.922655 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:35.922679 (+ 24us) server_negotiation.cc:407] Connection header received
0923 14:36:35.922725 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:35.922730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:35.922888 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:35.923078 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:35.923641 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:35.923644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:35.923652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:35.923696 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:35.923736 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:35.923764 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:35.924634 (+ 870us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:35.924673 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 14:36:35.924727 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":120,"threads_started":1}
I20260923 14:36:35.926788 7944 tablet_service.cc:1467] Processing CreateTablet for tablet a9e8ccdef280430a91085a8bc18fbebe (DEFAULT_TABLE table=test-table [id=9c0b1a915f7f4b85adb0eedac4144fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:35.926970 7682 tablet_service.cc:1467] Processing CreateTablet for tablet a9e8ccdef280430a91085a8bc18fbebe (DEFAULT_TABLE table=test-table [id=9c0b1a915f7f4b85adb0eedac4144fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:35.927314 7813 tablet_service.cc:1467] Processing CreateTablet for tablet a9e8ccdef280430a91085a8bc18fbebe (DEFAULT_TABLE table=test-table [id=9c0b1a915f7f4b85adb0eedac4144fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:35.927699 7944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e8ccdef280430a91085a8bc18fbebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:35.927809 7682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e8ccdef280430a91085a8bc18fbebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:35.928100 7813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e8ccdef280430a91085a8bc18fbebe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:35.933423 8024 tablet_bootstrap.cc:492] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7: Bootstrap starting.
I20260923 14:36:35.934460 8026 tablet_bootstrap.cc:492] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: Bootstrap starting.
I20260923 14:36:35.935375 8024 tablet_bootstrap.cc:654] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:35.935973 8026 tablet_bootstrap.cc:654] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:35.936054 8024 log.cc:824] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:35.936642 8026 log.cc:824] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:35.937237 8025 tablet_bootstrap.cc:492] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: Bootstrap starting.
I20260923 14:36:35.937815 8024 tablet_bootstrap.cc:492] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7: No bootstrap required, opened a new log
I20260923 14:36:35.937994 8024 ts_tablet_manager.cc:1397] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:36:35.938452 8026 tablet_bootstrap.cc:492] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: No bootstrap required, opened a new log
I20260923 14:36:35.938614 8026 ts_tablet_manager.cc:1397] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260923 14:36:35.939086 8025 tablet_bootstrap.cc:654] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:35.939749 8025 log.cc:824] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:35.941218 8024 raft_consensus.cc:348] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 } } peers { permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 } } peers { permanent_uuid: "c0c718b0b791493494b783aa793504b7" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42375 } }
I20260923 14:36:35.941408 8024 raft_consensus.cc:374] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:35.941429 8025 tablet_bootstrap.cc:492] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: No bootstrap required, opened a new log
I20260923 14:36:35.941453 8024 raft_consensus.cc:729] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0c718b0b791493494b783aa793504b7, State: Initialized, Role: FOLLOWER
I20260923 14:36:35.941562 8025 ts_tablet_manager.cc:1397] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 14:36:35.941844 8024 consensus_queue.cc:260] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 } } peers { permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 } } peers { permanent_uuid: "c0c718b0b791493494b783aa793504b7" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42375 } }
I20260923 14:36:35.942371 7748 heartbeater.cc:503] Master 127.2.218.62:44385 was elected leader, sending a full tablet report...
I20260923 14:36:35.942266 8026 raft_consensus.cc:348] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 } } peers { permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 } } peers { permanent_uuid: "c0c718b0b791493494b783aa793504b7" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42375 } }
I20260923 14:36:35.942458 8026 raft_consensus.cc:374] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:35.942498 8026 raft_consensus.cc:729] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 517c428d5da4412ca0dfa3bea4cf5584, State: Initialized, Role: FOLLOWER
I20260923 14:36:35.942732 8024 ts_tablet_manager.cc:1428] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 14:36:35.942838 8026 consensus_queue.cc:260] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 } } peers { permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 } } peers { permanent_uuid: "c0c718b0b791493494b783aa793504b7" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42375 } }
I20260923 14:36:35.943400 7879 heartbeater.cc:503] Master 127.2.218.62:44385 was elected leader, sending a full tablet report...
I20260923 14:36:35.943637 8026 ts_tablet_manager.cc:1428] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:36:35.945116 8025 raft_consensus.cc:348] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 } } peers { permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 } } peers { permanent_uuid: "c0c718b0b791493494b783aa793504b7" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42375 } }
I20260923 14:36:35.945340 8025 raft_consensus.cc:374] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:35.945399 8025 raft_consensus.cc:729] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beea9d5ce054403c9ea23df31b130181, State: Initialized, Role: FOLLOWER
I20260923 14:36:35.945742 8025 consensus_queue.cc:260] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 } } peers { permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 } } peers { permanent_uuid: "c0c718b0b791493494b783aa793504b7" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42375 } }
I20260923 14:36:35.946398 8010 heartbeater.cc:503] Master 127.2.218.62:44385 was elected leader, sending a full tablet report...
I20260923 14:36:35.946568 8025 ts_tablet_manager.cc:1428] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260923 14:36:35.989298 7880 tablet.cc:2369] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:36.058405 8030 raft_consensus.cc:482] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:36.058566 8030 raft_consensus.cc:504] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 } } peers { permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 } } peers { permanent_uuid: "c0c718b0b791493494b783aa793504b7" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42375 } }
I20260923 14:36:36.059635 8030 leader_election.cc:288] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers beea9d5ce054403c9ea23df31b130181 (127.2.218.3:43015), 517c428d5da4412ca0dfa3bea4cf5584 (127.2.218.2:37537)
I20260923 14:36:36.061396 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.059860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:43015 (local address 127.2.218.1:56675)
0923 14:36:36.059976 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:36.059990 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:36.060060 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:36.060309 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:36.060313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:36.060326 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:36.060888 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:36:36.060889 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:36.060898 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:36:36.060920 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:36.061151 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:36.061154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:36.061157 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:36:36.061231 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 14:36:36.061263 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 14:36:36.061419 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.059943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:56675 (local address 127.2.218.3:43015)
0923 14:36:36.060087 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:36.060091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:36.060106 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:36:36.060148 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:36.060153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:36.060188 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:36.060271 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:36.061028 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:36.061031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:36.061034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:36.061044 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:36.061061 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:36.061069 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:36.061164 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:36.061251 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 14:36:36.061284 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:36:36.061918 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.060222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:60623 (local address 127.2.218.2:37537)
0923 14:36:36.060338 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:36.060342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:36.060588 (+ 246us) server_negotiation.cc:407] Connection header received
0923 14:36:36.060677 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:36.060682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:36.060718 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:36.060790 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:36.061579 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:36.061582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:36.061584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:36.061597 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:36.061615 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:36.061623 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:36.061697 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:36.061789 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 14:36:36.061822 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:36:36.061932 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.060097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:37537 (local address 127.2.218.1:60623)
0923 14:36:36.060493 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:36.060508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:36.060591 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:36.060825 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:36.060827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:36.060849 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:36.061402 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:36:36.061404 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:36.061412 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:36:36.061438 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:36.061722 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:36.061725 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:36.061728 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:36:36.061791 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 14:36:36.061819 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":139,"threads_started":1}
I20260923 14:36:36.062037 7964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e8ccdef280430a91085a8bc18fbebe" candidate_uuid: "c0c718b0b791493494b783aa793504b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beea9d5ce054403c9ea23df31b130181" is_pre_election: true
I20260923 14:36:36.062348 7964 raft_consensus.cc:2454] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0c718b0b791493494b783aa793504b7 in term 0.
I20260923 14:36:36.062675 7833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e8ccdef280430a91085a8bc18fbebe" candidate_uuid: "c0c718b0b791493494b783aa793504b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "517c428d5da4412ca0dfa3bea4cf5584" is_pre_election: true
I20260923 14:36:36.062780 7635 leader_election.cc:302] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beea9d5ce054403c9ea23df31b130181, c0c718b0b791493494b783aa793504b7; no voters:
I20260923 14:36:36.062922 7833 raft_consensus.cc:2454] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0c718b0b791493494b783aa793504b7 in term 0.
I20260923 14:36:36.063031 8030 raft_consensus.cc:2781] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:36.063117 8030 raft_consensus.cc:482] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:36.063164 8030 raft_consensus.cc:3037] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:36.064174 8030 raft_consensus.cc:504] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 } } peers { permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 } } peers { permanent_uuid: "c0c718b0b791493494b783aa793504b7" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42375 } }
I20260923 14:36:36.064462 8030 leader_election.cc:288] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [CANDIDATE]: Term 1 election: Requested vote from peers beea9d5ce054403c9ea23df31b130181 (127.2.218.3:43015), 517c428d5da4412ca0dfa3bea4cf5584 (127.2.218.2:37537)
I20260923 14:36:36.064749 7964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e8ccdef280430a91085a8bc18fbebe" candidate_uuid: "c0c718b0b791493494b783aa793504b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "beea9d5ce054403c9ea23df31b130181"
I20260923 14:36:36.064800 7833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e8ccdef280430a91085a8bc18fbebe" candidate_uuid: "c0c718b0b791493494b783aa793504b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "517c428d5da4412ca0dfa3bea4cf5584"
I20260923 14:36:36.064849 7964 raft_consensus.cc:3037] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:36.064916 7833 raft_consensus.cc:3037] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:36.065755 7964 raft_consensus.cc:2454] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c718b0b791493494b783aa793504b7 in term 1.
I20260923 14:36:36.065845 7833 raft_consensus.cc:2454] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0c718b0b791493494b783aa793504b7 in term 1.
I20260923 14:36:36.066004 7635 leader_election.cc:302] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: beea9d5ce054403c9ea23df31b130181, c0c718b0b791493494b783aa793504b7; no voters:
I20260923 14:36:36.066188 8030 raft_consensus.cc:2781] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:36.066392 8030 raft_consensus.cc:686] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [term 1 LEADER]: Becoming Leader. State: Replica: c0c718b0b791493494b783aa793504b7, State: Running, Role: LEADER
I20260923 14:36:36.066673 8030 consensus_queue.cc:237] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 } } peers { permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 } } peers { permanent_uuid: "c0c718b0b791493494b783aa793504b7" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42375 } }
I20260923 14:36:36.069339 7558 catalog_manager.cc:5495] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0c718b0b791493494b783aa793504b7 (127.2.218.1). New cstate: current_term: 1 leader_uuid: "c0c718b0b791493494b783aa793504b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0c718b0b791493494b783aa793504b7" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42375 } health_report { overall_health: HEALTHY } } }
I20260923 14:36:36.082125 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.080653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.2.218.1:42375)
0923 14:36:36.080752 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:36.080756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:36.080790 (+ 34us) server_negotiation.cc:407] Connection header received
0923 14:36:36.080955 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:36.080957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:36.081013 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:36.081087 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:36.081725 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:36.081728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:36.081730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:36.081740 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:36.081774 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:36.081788 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:36.081876 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:36.081951 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 14:36:36.081982 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:36:36.087085 7833 raft_consensus.cc:1261] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584 [term 1 FOLLOWER]: Refusing update from remote peer c0c718b0b791493494b783aa793504b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:36.087126 7964 raft_consensus.cc:1261] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181 [term 1 FOLLOWER]: Refusing update from remote peer c0c718b0b791493494b783aa793504b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:36.087674 8030 consensus_queue.cc:1035] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beea9d5ce054403c9ea23df31b130181" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:36.087846 8034 consensus_queue.cc:1035] T a9e8ccdef280430a91085a8bc18fbebe P c0c718b0b791493494b783aa793504b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "517c428d5da4412ca0dfa3bea4cf5584" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 37537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:36.092762 8037 mvcc.cc:205] Tried to move back new op lower bound from 7332553507166773248 to 7332553507091255296. Current Snapshot: MvccSnapshot[applied={T|T < 7332553507166773248}]
I20260923 14:36:36.096235 8039 mvcc.cc:205] Tried to move back new op lower bound from 7332553507166773248 to 7332553507091255296. Current Snapshot: MvccSnapshot[applied={T|T < 7332553507166773248}]
I20260923 14:36:36.101475 7558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:36.101639 7558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:36.103668 7558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce49aa068500408291c9e190fde17837: Sending DeleteTablet for 3 replicas of tablet a9e8ccdef280430a91085a8bc18fbebe
I20260923 14:36:36.104242 7944 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e8ccdef280430a91085a8bc18fbebe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:36 UTC) from {username='slave'} at 127.0.0.1:40110
I20260923 14:36:36.104514 7813 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e8ccdef280430a91085a8bc18fbebe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:36 UTC) from {username='slave'} at 127.0.0.1:57366
I20260923 14:36:36.104794 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 7620
I20260923 14:36:36.104908 7682 tablet_service.cc:1514] Processing DeleteTablet for tablet a9e8ccdef280430a91085a8bc18fbebe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:36 UTC) from {username='slave'} at 127.0.0.1:38516
I20260923 14:36:36.105508 8047 tablet_replica.cc:330] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: stopping tablet replica
I20260923 14:36:36.105695 8047 raft_consensus.cc:2229] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:36.105861 8047 raft_consensus.cc:2258] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:36.107275 8047 ts_tablet_manager.cc:1905] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:36.107496 8046 tablet_replica.cc:330] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: stopping tablet replica
I20260923 14:36:36.107725 8046 raft_consensus.cc:2229] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:36.107945 8046 raft_consensus.cc:2258] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:36.109078 8046 ts_tablet_manager.cc:1905] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:36.109767 8047 ts_tablet_manager.cc:1918] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:36.109851 8047 log.cc:1196] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/a9e8ccdef280430a91085a8bc18fbebe
I20260923 14:36:36.110162 8047 ts_tablet_manager.cc:1939] T a9e8ccdef280430a91085a8bc18fbebe P 517c428d5da4412ca0dfa3bea4cf5584: Deleting consensus metadata
W20260923 14:36:36.110723 7544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:42375 (error 108)
I20260923 14:36:36.111400 7544 catalog_manager.cc:4841] TS 517c428d5da4412ca0dfa3bea4cf5584 (127.2.218.2:37537): tablet a9e8ccdef280430a91085a8bc18fbebe (table test-table [id=9c0b1a915f7f4b85adb0eedac4144fde]) successfully deleted
I20260923 14:36:36.111442 8046 ts_tablet_manager.cc:1918] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:36.111455 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 7751
I20260923 14:36:36.111509 8046 log.cc:1196] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/a9e8ccdef280430a91085a8bc18fbebe
I20260923 14:36:36.111550 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.111274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:42375 (local address 127.0.0.1:38532)
0923 14:36:36.111407 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:36.111472 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:42375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 14:36:36.111739 8046 ts_tablet_manager.cc:1939] T a9e8ccdef280430a91085a8bc18fbebe P beea9d5ce054403c9ea23df31b130181: Deleting consensus metadata
W20260923 14:36:36.111958 7544 catalog_manager.cc:4584] TS c0c718b0b791493494b783aa793504b7 (127.2.218.1:42375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9e8ccdef280430a91085a8bc18fbebe: Network error: Client connection negotiation failed: client connection to 127.2.218.1:42375: connect: Connection refused (error 111)
I20260923 14:36:36.112604 7544 catalog_manager.cc:4841] TS beea9d5ce054403c9ea23df31b130181 (127.2.218.3:43015): tablet a9e8ccdef280430a91085a8bc18fbebe (table test-table [id=9c0b1a915f7f4b85adb0eedac4144fde]) successfully deleted
I20260923 14:36:36.116634 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 7882
I20260923 14:36:36.120851 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 7529
2026-09-23T14:36:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (881 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 14:36:36.140239 2920 mini_oidc.cc:152] Starting JWKS server
I20260923 14:36:36.141526 2920 webserver.cc:460] Webserver started at https://127.0.0.1:39529/ using document root <none> and password file <none>
I20260923 14:36:36.141717 2920 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 14:36:36.141963 2920 webserver.cc:460] Webserver started at http://127.0.0.1:39893/ using document root <none> and password file <none>
2026-09-23T14:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:36Z Disabled control of system clock
I20260923 14:36:36.149307 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:41533
--webserver_interface=127.2.218.62
--webserver_port=0
--builtin_ntp_servers=127.2.218.20:46491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:41533
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39529/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/testchainca.cert with env {}
W20260923 14:36:36.240224 8054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:36.240406 8054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:36.240435 8054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:36.243896 8054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:36.243937 8054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:36.243950 8054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:36.243963 8054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:36.243976 8054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 14:36:36.243988 8054 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 14:36:36.244000 8054 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39529/jwks.json
I20260923 14:36:36.248277 8054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:46491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:41533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39529/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:36.248956 8054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:36.249667 8054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:36.256378 8062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:36.256502 8054 server_base.cc:1034] running on GCE node
W20260923 14:36:36.256534 8060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:36.256409 8059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:36.257123 8054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:36.257931 8054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:36.259085 8054 hybrid_clock.cc:648] HybridClock initialized: now 1790174196259062 us; error 32 us; skew 500 ppm
I20260923 14:36:36.260564 8054 webserver.cc:460] Webserver started at http://127.2.218.62:33757/ using document root <none> and password file <none>
I20260923 14:36:36.261039 8054 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:36.261091 8054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:36.261330 8054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:36.263229 8054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "44c0b200eb044e51bc90238fa204f067"
format_stamp: "Formatted at 2026-09-23 14:36:36 on dist-test-slave-zl47"
server_key: "b1e21e798836b0946e45be36c5fc0e33"
server_key_iv: "81086ee340eb622a94f8741311c52c4d"
server_key_version: "encryptionkey@0"
I20260923 14:36:36.263702 8054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "44c0b200eb044e51bc90238fa204f067"
format_stamp: "Formatted at 2026-09-23 14:36:36 on dist-test-slave-zl47"
server_key: "b1e21e798836b0946e45be36c5fc0e33"
server_key_iv: "81086ee340eb622a94f8741311c52c4d"
server_key_version: "encryptionkey@0"
I20260923 14:36:36.266604 8054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:36:36.268483 8068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:36.269454 8054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:36.269572 8054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "44c0b200eb044e51bc90238fa204f067"
format_stamp: "Formatted at 2026-09-23 14:36:36 on dist-test-slave-zl47"
server_key: "b1e21e798836b0946e45be36c5fc0e33"
server_key_iv: "81086ee340eb622a94f8741311c52c4d"
server_key_version: "encryptionkey@0"
I20260923 14:36:36.269665 8054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:36.293711 8054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:36.294255 8054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:36.294415 8054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:36.300400 8054 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:41533
I20260923 14:36:36.300405 8120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:41533 every 8 connection(s)
I20260923 14:36:36.301148 8054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:36.303403 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 8054
I20260923 14:36:36.303524 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:36.303767 2920 external_mini_cluster.cc:1428] Setting key 9bc83453a21c9abe446f941cefd62419
I20260923 14:36:36.303982 8121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:36.309136 8121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067: Bootstrap starting.
I20260923 14:36:36.311096 8121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:36.311717 8121 log.cc:824] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:36.311726 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.304838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.2.218.62:41533)
0923 14:36:36.305381 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:36.305393 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:36.305421 (+ 28us) server_negotiation.cc:407] Connection header received
0923 14:36:36.306236 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:36.306252 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:36.306578 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:36.306849 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:36.307196 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.308036 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.308899 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.309403 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.309556 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:36.310365 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:36.310369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:36.310379 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:36.310678 (+ 299us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:36.310723 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:36.310739 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:36.310834 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:36.310931 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 14:36:36.311159 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260923 14:36:36.313437 8121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067: No bootstrap required, opened a new log
I20260923 14:36:36.314124 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:36:36.316716 8121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0b200eb044e51bc90238fa204f067" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41533 } }
I20260923 14:36:36.316936 8121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:36.316987 8121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c0b200eb044e51bc90238fa204f067, State: Initialized, Role: FOLLOWER
I20260923 14:36:36.317520 8121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0b200eb044e51bc90238fa204f067" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41533 } }
I20260923 14:36:36.317654 8121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:36.317700 8121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:36.317766 8121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:36.318506 8121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0b200eb044e51bc90238fa204f067" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41533 } }
I20260923 14:36:36.318761 8121 leader_election.cc:302] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44c0b200eb044e51bc90238fa204f067; no voters:
I20260923 14:36:36.318997 8121 leader_election.cc:288] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:36.319101 8126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:36.319371 8126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [term 1 LEADER]: Becoming Leader. State: Replica: 44c0b200eb044e51bc90238fa204f067, State: Running, Role: LEADER
I20260923 14:36:36.319619 8126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0b200eb044e51bc90238fa204f067" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41533 } }
I20260923 14:36:36.320094 8121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:36.321177 8128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44c0b200eb044e51bc90238fa204f067. Latest consensus state: current_term: 1 leader_uuid: "44c0b200eb044e51bc90238fa204f067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0b200eb044e51bc90238fa204f067" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41533 } } }
I20260923 14:36:36.321301 8128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:36.321672 8135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:36.323854 8135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:36.321627 8127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44c0b200eb044e51bc90238fa204f067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c0b200eb044e51bc90238fa204f067" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41533 } } }
I20260923 14:36:36.325287 8127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:36.328016 8135 catalog_manager.cc:1347] Generated new cluster ID: d61b5ae9f9ed4b539a5b0c034495b1d6
I20260923 14:36:36.328066 8135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:36.341593 8135 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:36.342326 8135 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:36.347290 8135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44c0b200eb044e51bc90238fa204f067: Generated new TSK 0
I20260923 14:36:36.347755 8135 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 14:36:36.381319 8151 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 14:36:36.381414 8151 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260923 14:36:36.382544 8150 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:36.361504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:41533 (local address 127.0.0.1:39406)
0923 14:36:36.361832 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:36.361858 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:36.361974 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:36.362298 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:36.362301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:36.362308 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 14:36:36.362431 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:36.362438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.363059 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.363062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:36.363934 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:36.363944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.364326 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.364330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:36.364418 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:36.364464 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 14:36:36.381590 (+ 17126us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 14:36:36.381724 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.218.62:41533: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":113,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.218.62:41533: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 14:36:36.382952 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.361620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.2.218.62:41533)
0923 14:36:36.361722 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:36.361726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:36.361927 (+ 201us) server_negotiation.cc:407] Connection header received
0923 14:36:36.362142 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:36.362144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:36.362189 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:36.362256 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0923 14:36:36.362531 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.362959 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.364038 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.364227 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.364291 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:36.364605 (+ 314us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 14:36:36.381406 (+ 16801us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 14:36:36.382841 (+ 1435us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:36:36.383000 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 8054
W20260923 14:36:36.383000 8124 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
2026-09-23T14:36:36Z chronyd exiting
I20260923 14:36:36.419049 2920 test_util.cc:175] -----------------------------------------------
I20260923 14:36:36.419097 2920 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790174173335390-2920-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (280 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 14:36:36.419878 2920 mini_oidc.cc:152] Starting JWKS server
I20260923 14:36:36.420619 2920 webserver.cc:460] Webserver started at https://127.0.0.1:38393/ using document root <none> and password file <none>
I20260923 14:36:36.420689 2920 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 14:36:36.420933 2920 webserver.cc:460] Webserver started at http://127.0.0.1:33211/ using document root <none> and password file <none>
2026-09-23T14:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:36Z Disabled control of system clock
I20260923 14:36:36.428146 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:46203
--webserver_interface=127.2.218.62
--webserver_port=0
--builtin_ntp_servers=127.2.218.20:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:46203
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38393/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/testchainca.cert with env {}
W20260923 14:36:36.521281 8157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:36.521482 8157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:36.521530 8157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:36.524870 8157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:36.524910 8157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:36.524923 8157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:36.524937 8157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:36.524950 8157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 14:36:36.524962 8157 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 14:36:36.524973 8157 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38393/jwks.json
I20260923 14:36:36.529289 8157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:46203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38393/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:36.529992 8157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:36.530735 8157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:36.537541 8165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:36.537684 8157 server_base.cc:1034] running on GCE node
W20260923 14:36:36.537541 8162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:36.537578 8163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:36.538305 8157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:36.539227 8157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:36.540405 8157 hybrid_clock.cc:648] HybridClock initialized: now 1790174196540383 us; error 35 us; skew 500 ppm
I20260923 14:36:36.541967 8157 webserver.cc:460] Webserver started at http://127.2.218.62:35323/ using document root <none> and password file <none>
I20260923 14:36:36.542505 8157 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:36.542598 8157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:36.542799 8157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:36.544498 8157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "8a5dac644970406aa95cd296159b638c"
format_stamp: "Formatted at 2026-09-23 14:36:36 on dist-test-slave-zl47"
server_key: "37d4f6797b14ca52b88688d264776ded"
server_key_iv: "8378ed19ab0fe9436ba27323e76f4a2d"
server_key_version: "encryptionkey@0"
I20260923 14:36:36.545014 8157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "8a5dac644970406aa95cd296159b638c"
format_stamp: "Formatted at 2026-09-23 14:36:36 on dist-test-slave-zl47"
server_key: "37d4f6797b14ca52b88688d264776ded"
server_key_iv: "8378ed19ab0fe9436ba27323e76f4a2d"
server_key_version: "encryptionkey@0"
I20260923 14:36:36.548143 8157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:36.550072 8171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:36.551062 8157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:36.551189 8157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "8a5dac644970406aa95cd296159b638c"
format_stamp: "Formatted at 2026-09-23 14:36:36 on dist-test-slave-zl47"
server_key: "37d4f6797b14ca52b88688d264776ded"
server_key_iv: "8378ed19ab0fe9436ba27323e76f4a2d"
server_key_version: "encryptionkey@0"
I20260923 14:36:36.551287 8157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:36.579604 8157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:36.580220 8157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:36.580395 8157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:36.586483 8223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:46203 every 8 connection(s)
I20260923 14:36:36.586481 8157 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:46203
I20260923 14:36:36.587291 8157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:36.589936 8224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:36.592278 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 8157
I20260923 14:36:36.592398 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:36.592643 2920 external_mini_cluster.cc:1428] Setting key 1dfedc53513ee07892aca2f84e5d47c7
I20260923 14:36:36.594730 8224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c: Bootstrap starting.
I20260923 14:36:36.596627 8224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:36.597370 8224 log.cc:824] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:36.599129 8224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c: No bootstrap required, opened a new log
I20260923 14:36:36.600332 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.593749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.2.218.62:46203)
0923 14:36:36.594309 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:36.594324 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:36.594354 (+ 30us) server_negotiation.cc:407] Connection header received
0923 14:36:36.594943 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:36.594958 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:36.595282 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:36.595518 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:36.595807 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.596593 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.597622 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.598016 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.598190 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:36.598870 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:36.598874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:36.598884 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:36.599244 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:36.599296 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:36.599315 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:36.599390 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:36.599535 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 14:36:36.599782 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":115,"threads_started":1}
I20260923 14:36:36.602173 8224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5dac644970406aa95cd296159b638c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46203 } }
I20260923 14:36:36.602361 8224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:36.602428 8224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5dac644970406aa95cd296159b638c, State: Initialized, Role: FOLLOWER
I20260923 14:36:36.602762 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:36:36.602829 8224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5dac644970406aa95cd296159b638c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46203 } }
I20260923 14:36:36.602949 8224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:36.602996 8224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:36.603068 8224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:36.603883 8224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5dac644970406aa95cd296159b638c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46203 } }
I20260923 14:36:36.604174 8224 leader_election.cc:302] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a5dac644970406aa95cd296159b638c; no voters:
I20260923 14:36:36.604470 8224 leader_election.cc:288] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:36.604504 8229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:36.604667 8229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [term 1 LEADER]: Becoming Leader. State: Replica: 8a5dac644970406aa95cd296159b638c, State: Running, Role: LEADER
I20260923 14:36:36.604921 8229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5dac644970406aa95cd296159b638c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46203 } }
I20260923 14:36:36.605504 8224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:36.606612 8230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a5dac644970406aa95cd296159b638c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5dac644970406aa95cd296159b638c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46203 } } }
I20260923 14:36:36.606729 8230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:36.606964 8231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a5dac644970406aa95cd296159b638c. Latest consensus state: current_term: 1 leader_uuid: "8a5dac644970406aa95cd296159b638c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5dac644970406aa95cd296159b638c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46203 } } }
I20260923 14:36:36.607052 8231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:36.607488 8238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:36.609650 8238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:36.613886 8238 catalog_manager.cc:1347] Generated new cluster ID: 5a38f26f190f4ca397a99e0e7947f233
I20260923 14:36:36.613948 8238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:36.633641 8238 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:36.634368 8238 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:36.639575 8238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a5dac644970406aa95cd296159b638c: Generated new TSK 0
I20260923 14:36:36.640067 8238 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 14:36:36.660277 8254 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 14:36:36.660346 8254 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 14:36:36.660807 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.647686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.2.218.62:46203)
0923 14:36:36.647791 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:36.647795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:36.647915 (+ 120us) server_negotiation.cc:407] Connection header received
0923 14:36:36.648011 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:36.648013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:36.648061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:36.648163 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0923 14:36:36.648326 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.649008 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.649739 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.650036 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.650154 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:36.650382 (+ 228us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 14:36:36.660521 (+ 10139us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 14:36:36.660705 (+ 184us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 14:36:36.660873 8227 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 14:36:36.660897 8253 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:36.647595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:46203 (local address 127.0.0.1:40436)
0923 14:36:36.647845 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:36.647855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:36.647931 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:36.648160 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:36.648162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:36.648167 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0923 14:36:36.648242 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:36.648245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.649102 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.649104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:36.649642 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:36.649649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.650179 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.650182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:36.650236 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:36.650254 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 14:36:36.660667 (+ 10413us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 14:36:36.660758 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.218.62:46203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38393/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20260923 14:36:36.661463 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 8157
2026-09-23T14:36:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (279 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 14:36:36.699230 2920 mini_oidc.cc:152] Starting JWKS server
I20260923 14:36:36.699960 2920 webserver.cc:460] Webserver started at https://127.0.0.1:46835/ using document root <none> and password file <none>
I20260923 14:36:36.700057 2920 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 14:36:36.700306 2920 webserver.cc:460] Webserver started at http://127.0.0.1:34459/ using document root <none> and password file <none>
2026-09-23T14:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:36Z Disabled control of system clock
I20260923 14:36:36.707777 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:43795
--webserver_interface=127.2.218.62
--webserver_port=0
--builtin_ntp_servers=127.2.218.20:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:43795
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46835/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 14:36:36.799598 8260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:36.799834 8260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:36.799897 8260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:36.803534 8260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:36.803613 8260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:36.803637 8260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:36.803668 8260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:36.803694 8260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 14:36:36.803725 8260 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 14:36:36.803753 8260 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46835/jwks.json
I20260923 14:36:36.807610 8260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:43795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:43795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46835/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:36.808415 8260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:36.809476 8260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:36.816337 8265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:36.816303 8268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:36.816310 8266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:36.816540 8260 server_base.cc:1034] running on GCE node
I20260923 14:36:36.817087 8260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:36.818233 8260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:36.819413 8260 hybrid_clock.cc:648] HybridClock initialized: now 1790174196819400 us; error 32 us; skew 500 ppm
I20260923 14:36:36.821022 8260 webserver.cc:460] Webserver started at http://127.2.218.62:36693/ using document root <none> and password file <none>
I20260923 14:36:36.821645 8260 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:36.821735 8260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:36.821952 8260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:36.823916 8260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "fe9b9f89d54c40888e6ace783de4e372"
format_stamp: "Formatted at 2026-09-23 14:36:36 on dist-test-slave-zl47"
server_key: "d5820b76a783647d4b95bd5c234bb278"
server_key_iv: "927fb3d3a7df33f8e147e4ac8ed3bfc7"
server_key_version: "encryptionkey@0"
I20260923 14:36:36.824476 8260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "fe9b9f89d54c40888e6ace783de4e372"
format_stamp: "Formatted at 2026-09-23 14:36:36 on dist-test-slave-zl47"
server_key: "d5820b76a783647d4b95bd5c234bb278"
server_key_iv: "927fb3d3a7df33f8e147e4ac8ed3bfc7"
server_key_version: "encryptionkey@0"
I20260923 14:36:36.827718 8260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:36.829859 8274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:36.830952 8260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:36.831087 8260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "fe9b9f89d54c40888e6ace783de4e372"
format_stamp: "Formatted at 2026-09-23 14:36:36 on dist-test-slave-zl47"
server_key: "d5820b76a783647d4b95bd5c234bb278"
server_key_iv: "927fb3d3a7df33f8e147e4ac8ed3bfc7"
server_key_version: "encryptionkey@0"
I20260923 14:36:36.831204 8260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:36.853933 8260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:36.854498 8260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:36.854682 8260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:36.860719 8260 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:43795
I20260923 14:36:36.860744 8326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:43795 every 8 connection(s)
I20260923 14:36:36.861522 8260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:36.861934 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 8260
I20260923 14:36:36.862048 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:36.862398 2920 external_mini_cluster.cc:1428] Setting key ffa8215c8da94e5761bf977609619852
I20260923 14:36:36.864594 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:36.870067 8327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372: Bootstrap starting.
I20260923 14:36:36.870401 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.863490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.2.218.62:43795)
0923 14:36:36.864020 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:36.864035 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:36.864066 (+ 31us) server_negotiation.cc:407] Connection header received
0923 14:36:36.864609 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:36.864623 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:36.865057 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:36.865318 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:36.865614 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.866532 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.867586 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.867977 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.868127 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:36.868852 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:36.868857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:36.868866 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:36.869195 (+ 329us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:36.869243 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:36.869261 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:36.869336 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:36.869491 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 14:36:36.869733 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":137,"threads_started":1}
I20260923 14:36:36.871923 8327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:36.872531 8327 log.cc:824] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:36.872653 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:36:36.874055 8327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372: No bootstrap required, opened a new log
I20260923 14:36:36.876770 8327 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b9f89d54c40888e6ace783de4e372" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43795 } }
I20260923 14:36:36.876959 8327 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:36.877028 8327 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe9b9f89d54c40888e6ace783de4e372, State: Initialized, Role: FOLLOWER
I20260923 14:36:36.877408 8327 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b9f89d54c40888e6ace783de4e372" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43795 } }
I20260923 14:36:36.877534 8327 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:36.877614 8327 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:36.877697 8327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:36.878450 8327 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b9f89d54c40888e6ace783de4e372" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43795 } }
I20260923 14:36:36.878733 8327 leader_election.cc:302] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe9b9f89d54c40888e6ace783de4e372; no voters:
I20260923 14:36:36.878973 8327 leader_election.cc:288] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:36.879082 8332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:36.879513 8332 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [term 1 LEADER]: Becoming Leader. State: Replica: fe9b9f89d54c40888e6ace783de4e372, State: Running, Role: LEADER
I20260923 14:36:36.879973 8332 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b9f89d54c40888e6ace783de4e372" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43795 } }
I20260923 14:36:36.879990 8327 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:36.881481 8334 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe9b9f89d54c40888e6ace783de4e372. Latest consensus state: current_term: 1 leader_uuid: "fe9b9f89d54c40888e6ace783de4e372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b9f89d54c40888e6ace783de4e372" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43795 } } }
I20260923 14:36:36.881479 8333 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe9b9f89d54c40888e6ace783de4e372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9b9f89d54c40888e6ace783de4e372" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43795 } } }
I20260923 14:36:36.881631 8333 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:36.881631 8334 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372 [sys.catalog]: This master's current role is: LEADER
W20260923 14:36:36.883919 8347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:36:36.883999 8347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:36:36.884063 8346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:36.884678 8346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:36.888921 8346 catalog_manager.cc:1347] Generated new cluster ID: 3e718162c3bc4a4abeaea2f7ea8e93ba
I20260923 14:36:36.888978 8346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:36.903194 8346 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:36.903900 8346 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:36.911142 8346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe9b9f89d54c40888e6ace783de4e372: Generated new TSK 0
I20260923 14:36:36.911651 8346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:36:36.923493 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:36.919216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.2.218.62:43795)
0923 14:36:36.919319 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:36.919323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:36.919450 (+ 127us) server_negotiation.cc:407] Connection header received
0923 14:36:36.919570 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:36.919572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:36.919610 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:36.919680 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0923 14:36:36.919901 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.920346 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.921484 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.921673 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.921748 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:36.922055 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 14:36:36.923209 (+ 1154us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46835/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 14:36:36.923392 (+ 183us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36626: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46835/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 14:36:36.923538 8330 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36626: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46835/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 14:36:36.925093 8356 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:36.919091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:43795 (local address 127.0.0.1:36626)
0923 14:36:36.919370 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:36.919382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:36.919478 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:36.919703 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:36.919705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:36.919712 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 14:36:36.919813 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:36.919818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.920454 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.920457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:36.921374 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:36.921384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:36.921787 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:36.921791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:36.921875 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:36.921901 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 14:36:36.923355 (+ 1454us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46835/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 14:36:36.924947 (+ 1592us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.218.62:43795: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46835/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":1430,"thread_start_us":102,"threads_started":1}
I20260923 14:36:36.925521 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 8260
E20260923 14:36:36.932476 8357 webserver.cc:533] Webserver: sslize failed:
E20260923 14:36:36.932526 8357 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T14:36:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (264 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T14:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:36Z Disabled control of system clock
I20260923 14:36:36.970646 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:44577
--webserver_interface=127.2.218.62
--webserver_port=0
--builtin_ntp_servers=127.2.218.20:45327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:44577
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/testchainca.cert with env {}
W20260923 14:36:37.066567 8361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:37.066766 8361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:37.066799 8361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:37.070287 8361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:37.070331 8361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:37.070348 8361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:37.070363 8361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:37.070379 8361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 14:36:37.070394 8361 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 14:36:37.070415 8361 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 14:36:37.074659 8361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:44577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:44577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:37.075368 8361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:37.076107 8361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:37.083060 8367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:37.083168 8361 server_base.cc:1034] running on GCE node
W20260923 14:36:37.083058 8366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:37.083091 8369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:37.083747 8361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:37.084673 8361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:37.085915 8361 hybrid_clock.cc:648] HybridClock initialized: now 1790174197085882 us; error 28 us; skew 500 ppm
I20260923 14:36:37.087461 8361 webserver.cc:460] Webserver started at http://127.2.218.62:38501/ using document root <none> and password file <none>
I20260923 14:36:37.087872 8361 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:37.087924 8361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:37.088054 8361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:37.089600 8361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "06381c9932974bada28f58ad673ea213"
format_stamp: "Formatted at 2026-09-23 14:36:37 on dist-test-slave-zl47"
server_key: "ddb24e8a507b2d05ec91de0f2829e0e1"
server_key_iv: "54a568cab1b0a14ac7d121b440437095"
server_key_version: "encryptionkey@0"
I20260923 14:36:37.090041 8361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "06381c9932974bada28f58ad673ea213"
format_stamp: "Formatted at 2026-09-23 14:36:37 on dist-test-slave-zl47"
server_key: "ddb24e8a507b2d05ec91de0f2829e0e1"
server_key_iv: "54a568cab1b0a14ac7d121b440437095"
server_key_version: "encryptionkey@0"
I20260923 14:36:37.092972 8361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260923 14:36:37.094944 8375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:37.095913 8361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:37.095997 8361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "06381c9932974bada28f58ad673ea213"
format_stamp: "Formatted at 2026-09-23 14:36:37 on dist-test-slave-zl47"
server_key: "ddb24e8a507b2d05ec91de0f2829e0e1"
server_key_iv: "54a568cab1b0a14ac7d121b440437095"
server_key_version: "encryptionkey@0"
I20260923 14:36:37.096065 8361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:37.114956 8361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:37.115391 8361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:37.115504 8361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:37.121104 8361 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:44577
I20260923 14:36:37.121105 8427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:44577 every 8 connection(s)
I20260923 14:36:37.121847 8361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:37.124272 8428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:37.124861 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 8361
I20260923 14:36:37.124975 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:37.125250 2920 external_mini_cluster.cc:1428] Setting key f79864a07a51072fc6bbf4250203cacb
I20260923 14:36:37.129244 8428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213: Bootstrap starting.
I20260923 14:36:37.131183 8428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:37.131774 8428 log.cc:824] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:37.132812 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:37.126456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.2.218.62:44577)
0923 14:36:37.126970 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:37.126982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:37.127009 (+ 27us) server_negotiation.cc:407] Connection header received
0923 14:36:37.127524 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:37.127537 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:37.127864 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:37.128139 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:37.128487 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:37.129282 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:37.130030 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:37.130467 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:37.130622 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:37.131331 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:37.131335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:37.131345 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:36:37.131647 (+ 302us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:37.131697 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:36:37.131713 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:37.131809 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:37.131933 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 14:36:37.132175 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260923 14:36:37.133445 8428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213: No bootstrap required, opened a new log
I20260923 14:36:37.135037 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:36:37.136699 8428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06381c9932974bada28f58ad673ea213" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44577 } }
I20260923 14:36:37.136870 8428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:37.136936 8428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06381c9932974bada28f58ad673ea213, State: Initialized, Role: FOLLOWER
I20260923 14:36:37.137274 8428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06381c9932974bada28f58ad673ea213" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44577 } }
I20260923 14:36:37.137405 8428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:37.137478 8428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:37.137574 8428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:37.138310 8428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06381c9932974bada28f58ad673ea213" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44577 } }
I20260923 14:36:37.138581 8428 leader_election.cc:302] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06381c9932974bada28f58ad673ea213; no voters:
I20260923 14:36:37.138815 8428 leader_election.cc:288] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:37.138908 8433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:37.139096 8433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [term 1 LEADER]: Becoming Leader. State: Replica: 06381c9932974bada28f58ad673ea213, State: Running, Role: LEADER
I20260923 14:36:37.139364 8433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06381c9932974bada28f58ad673ea213" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44577 } }
I20260923 14:36:37.139854 8428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:37.140686 8435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06381c9932974bada28f58ad673ea213. Latest consensus state: current_term: 1 leader_uuid: "06381c9932974bada28f58ad673ea213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06381c9932974bada28f58ad673ea213" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44577 } } }
I20260923 14:36:37.140789 8435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:37.141088 8434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06381c9932974bada28f58ad673ea213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06381c9932974bada28f58ad673ea213" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44577 } } }
I20260923 14:36:37.141248 8434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213 [sys.catalog]: This master's current role is: LEADER
W20260923 14:36:37.144181 8449 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:36:37.144245 8449 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:36:37.144408 8442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:37.145153 8442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:37.149446 8442 catalog_manager.cc:1347] Generated new cluster ID: 61e5a2c580024240816bd663031ea76a
I20260923 14:36:37.149497 8442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:37.158347 8442 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:37.159062 8442 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:37.166679 8442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06381c9932974bada28f58ad673ea213: Generated new TSK 0
I20260923 14:36:37.167207 8442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:36:37.293222 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:37.175055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.2.218.62:44577)
0923 14:36:37.175184 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:37.175189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:37.175395 (+ 206us) server_negotiation.cc:407] Connection header received
0923 14:36:37.175531 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:37.175533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:37.175576 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:37.175665 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0923 14:36:37.175894 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:37.176397 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:37.177498 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:37.177694 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:37.177771 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:37.178076 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 14:36:37.292850 (+114774us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 14:36:37.293093 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38102: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260923 14:36:37.293316 8431 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38102: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 14:36:37.293430 8457 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:37.175022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:44577 (local address 127.0.0.1:38102)
0923 14:36:37.175307 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:37.175321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:37.175420 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:37.175688 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:37.175691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:37.175698 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0923 14:36:37.175797 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:37.175807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:37.176505 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:37.176508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:37.177358 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:37.177367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:37.177818 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:37.177821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:37.177912 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:37.177937 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 14:36:37.293073 (+115136us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 14:36:37.293255 (+ 182us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.218.62:44577: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
I20260923 14:36:37.293975 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 8361
2026-09-23T14:36:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (348 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (101 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (101 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:37 dist-test-slave-zl47 krb5kdc[8469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:37 dist-test-slave-zl47 krb5kdc[8469](info): set up 2 sockets
Sep 23 14:36:37 dist-test-slave-zl47 krb5kdc[8469](info): commencing operation
krb5kdc: starting...
W20260923 14:36:38.438906 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:38 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:38.549043 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:38309
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:38309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:38.649106 8482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:38.649322 8482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:38.649362 8482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:38.653131 8482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:38.653179 8482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:38.653198 8482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:38.653216 8482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:38.653232 8482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:38.657939 8482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:38309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:38309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:38.658809 8482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:38.659638 8482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:38.667382 8490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:38.667397 8488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:38.667402 8487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:38.667678 8482 server_base.cc:1034] running on GCE node
I20260923 14:36:38.668203 8482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:38.669129 8482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:38.670341 8482 hybrid_clock.cc:648] HybridClock initialized: now 1790174198670326 us; error 34 us; skew 500 ppm
Sep 23 14:36:38 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:38.672879 8482 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:38.673655 8482 webserver.cc:460] Webserver started at http://127.2.218.62:45431/ using document root <none> and password file <none>
I20260923 14:36:38.674060 8482 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:38.674168 8482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:38.674319 8482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:38.675949 8482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "1d652120729742ccbaaadc8576023fa9"
format_stamp: "Formatted at 2026-09-23 14:36:38 on dist-test-slave-zl47"
server_key: "69a82ff26a15d7cb87e811b449e98a4a"
server_key_iv: "d8b04429ec09ed6d72dd50300e5e7558"
server_key_version: "encryptionkey@0"
I20260923 14:36:38.676407 8482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "1d652120729742ccbaaadc8576023fa9"
format_stamp: "Formatted at 2026-09-23 14:36:38 on dist-test-slave-zl47"
server_key: "69a82ff26a15d7cb87e811b449e98a4a"
server_key_iv: "d8b04429ec09ed6d72dd50300e5e7558"
server_key_version: "encryptionkey@0"
I20260923 14:36:38.679406 8482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260923 14:36:38.681357 8497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:38.682482 8482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:38.682564 8482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "1d652120729742ccbaaadc8576023fa9"
format_stamp: "Formatted at 2026-09-23 14:36:38 on dist-test-slave-zl47"
server_key: "69a82ff26a15d7cb87e811b449e98a4a"
server_key_iv: "d8b04429ec09ed6d72dd50300e5e7558"
server_key_version: "encryptionkey@0"
I20260923 14:36:38.682638 8482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:38.692734 8482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:38.695331 8482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:38.695457 8482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:38.701776 8482 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:38309
I20260923 14:36:38.701805 8549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:38309 every 8 connection(s)
I20260923 14:36:38.702674 8482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:38.703787 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 8482
I20260923 14:36:38.703871 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:38.704113 2920 external_mini_cluster.cc:1428] Setting key 438205d8403ffde1adc23b9e63c3a060
I20260923 14:36:38.705355 8550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:38.711128 8550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9: Bootstrap starting.
Sep 23 14:36:38 dist-test-slave-zl47 krb5kdc[8469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:38.713253 8550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:38.714046 8550 log.cc:824] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:38.715816 8550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9: No bootstrap required, opened a new log
I20260923 14:36:38.717131 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:38.705273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.2.218.62:38309)
0923 14:36:38.705657 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:38.705674 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:38.705723 (+ 49us) server_negotiation.cc:407] Connection header received
0923 14:36:38.706489 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:38.706508 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:38.706880 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:38.707167 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:38.707769 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:38.709078 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:38.709899 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:38.710515 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:38.710653 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:38.712799 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:38.712805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:38.712819 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:38.712849 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:38.714515 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:38.714890 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:38.714895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:38.714900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:38.715012 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:38.715258 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:38.715261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:38.715262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:38.715646 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:38.715800 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:38.716020 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 14:36:38.716312 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":136,"threads_started":1}
I20260923 14:36:38.719436 8550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d652120729742ccbaaadc8576023fa9" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38309 } }
I20260923 14:36:38.719600 8550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:38.719628 8550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d652120729742ccbaaadc8576023fa9, State: Initialized, Role: FOLLOWER
I20260923 14:36:38.719907 8550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d652120729742ccbaaadc8576023fa9" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38309 } }
I20260923 14:36:38.719995 8550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:38.720034 8550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:38.720084 8550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:38.720690 8550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d652120729742ccbaaadc8576023fa9" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38309 } }
I20260923 14:36:38.720937 8550 leader_election.cc:302] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d652120729742ccbaaadc8576023fa9; no voters:
I20260923 14:36:38.721127 8550 leader_election.cc:288] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:38.721240 8555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:38.721443 8555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [term 1 LEADER]: Becoming Leader. State: Replica: 1d652120729742ccbaaadc8576023fa9, State: Running, Role: LEADER
I20260923 14:36:38.721724 8555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d652120729742ccbaaadc8576023fa9" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38309 } }
I20260923 14:36:38.722224 8550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:38.723229 8557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d652120729742ccbaaadc8576023fa9. Latest consensus state: current_term: 1 leader_uuid: "1d652120729742ccbaaadc8576023fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d652120729742ccbaaadc8576023fa9" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38309 } } }
I20260923 14:36:38.723358 8557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:38.723645 8556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d652120729742ccbaaadc8576023fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d652120729742ccbaaadc8576023fa9" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38309 } } }
I20260923 14:36:38.723737 8556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:38.723752 8564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:38.726557 8564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:38.731165 8564 catalog_manager.cc:1347] Generated new cluster ID: 38803cb7ed0842808136dbd3f665e61e
I20260923 14:36:38.731258 8564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:38.755894 8564 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:38.756655 8564 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:38.770653 8564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d652120729742ccbaaadc8576023fa9: Generated new TSK 0
I20260923 14:36:38.771291 8564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:36:38.807108 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38309
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:38.907066 8576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:38.907328 8576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:38.907404 8576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:38.911180 8576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:38.911255 8576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:38.911338 8576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:38.915934 8576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:38.916801 8576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:38.917682 8576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:38.925599 8581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:38.925621 8582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:38.925621 8584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:38.925737 8576 server_base.cc:1034] running on GCE node
I20260923 14:36:38.926180 8576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:38.926667 8576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:38.927824 8576 hybrid_clock.cc:648] HybridClock initialized: now 1790174198927807 us; error 27 us; skew 500 ppm
Sep 23 14:36:38 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:38.930330 8576 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:36:38.931057 8576 webserver.cc:460] Webserver started at http://127.2.218.1:41355/ using document root <none> and password file <none>
I20260923 14:36:38.931509 8576 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:38.931592 8576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:38.931792 8576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:38.933442 8576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "029ec4bc35304fc18d3786a3b21b73fc"
format_stamp: "Formatted at 2026-09-23 14:36:38 on dist-test-slave-zl47"
server_key: "8ef0bb1dba0064739304504c2f05441f"
server_key_iv: "6838b000b96bdfcca7db3c93f4b77987"
server_key_version: "encryptionkey@0"
I20260923 14:36:38.933928 8576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "029ec4bc35304fc18d3786a3b21b73fc"
format_stamp: "Formatted at 2026-09-23 14:36:38 on dist-test-slave-zl47"
server_key: "8ef0bb1dba0064739304504c2f05441f"
server_key_iv: "6838b000b96bdfcca7db3c93f4b77987"
server_key_version: "encryptionkey@0"
I20260923 14:36:38.937052 8576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260923 14:36:38.939261 8591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:38.940263 8576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:38.940384 8576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "029ec4bc35304fc18d3786a3b21b73fc"
format_stamp: "Formatted at 2026-09-23 14:36:38 on dist-test-slave-zl47"
server_key: "8ef0bb1dba0064739304504c2f05441f"
server_key_iv: "6838b000b96bdfcca7db3c93f4b77987"
server_key_version: "encryptionkey@0"
I20260923 14:36:38.940478 8576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:38.956424 8576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:38.958801 8576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:38.958993 8576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:38.959456 8576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:38.960146 8576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:38.960214 8576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:38.960274 8576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:38.960322 8576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:38.968942 8576 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:34195
I20260923 14:36:38.968968 8704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:34195 every 8 connection(s)
I20260923 14:36:38.969640 8576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:38.972621 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 8576
I20260923 14:36:38.972764 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:38.973125 2920 external_mini_cluster.cc:1428] Setting key a4da9137902a4e59b92e7a66052f6e35
Sep 23 14:36:38 dist-test-slave-zl47 krb5kdc[8469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
I20260923 14:36:38.983582 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:38.971354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:58287 (local address 127.2.218.62:38309)
0923 14:36:38.971521 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:38.971526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:38.972231 (+ 705us) server_negotiation.cc:407] Connection header received
0923 14:36:38.973870 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:38.973874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:38.973928 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:38.974018 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:38.975394 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:38.975925 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:38.977073 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:38.977296 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:38.977397 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:38.979986 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:38.979992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:38.979997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:38.980029 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:38.981539 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:38.982158 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:38.982160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:38.982161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:38.982229 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:38.982541 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:38.982543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:38.982544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:38.982698 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:38.982764 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:38.983354 (+ 590us) server_negotiation.cc:299] Negotiation successful
0923 14:36:38.983466 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:36:38.984166 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:38.971682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38309 (local address 127.2.218.1:58287)
0923 14:36:38.972083 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:38.972121 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:38.973612 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:38.974199 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:38.974205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:38.974676 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:38.975247 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:38.975267 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:38.976090 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:38.976093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:38.976928 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:38.976938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:38.977429 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:38.977433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:38.977577 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:38.978053 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:38.978066 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:38.979760 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:38.981663 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:38.981670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:38.981684 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:38.981990 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:38.982325 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:38.982329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:38.982331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:38.982458 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:38.982780 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:38.982784 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:38.982975 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 14:36:38.983138 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 14:36:38.983382 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
I20260923 14:36:38.985443 8705 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38309
I20260923 14:36:38.985642 8705 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:38.986058 8705 heartbeater.cc:511] Master 127.2.218.62:38309 requested a full tablet report, sending...
I20260923 14:36:38.987636 8514 ts_manager.cc:194] Registered new tserver with Master: 029ec4bc35304fc18d3786a3b21b73fc (127.2.218.1:34195)
I20260923 14:36:38.989279 8514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:58287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:36:38.999505 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38309
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:39.096127 8710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:39.096331 8710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:39.096364 8710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:39.099664 8710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:39.099711 8710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:39.099840 8710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:39.104272 8710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:39.105146 8710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:39.106005 8710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:39.114161 8716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:39.114220 8715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:39.114171 8718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:39.114349 8710 server_base.cc:1034] running on GCE node
I20260923 14:36:39.114686 8710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:39.115233 8710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:39.116413 8710 hybrid_clock.cc:648] HybridClock initialized: now 1790174199116396 us; error 38 us; skew 500 ppm
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:39.118757 8710 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:36:39.119467 8710 webserver.cc:460] Webserver started at http://127.2.218.2:38171/ using document root <none> and password file <none>
I20260923 14:36:39.119860 8710 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:39.119910 8710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:39.120036 8710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:39.121567 8710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "3ef5f42c4ba7428cb4abebdaaf7acc80"
format_stamp: "Formatted at 2026-09-23 14:36:39 on dist-test-slave-zl47"
server_key: "67b92c59f38c8baa781c3d8e91d173c2"
server_key_iv: "763b6ceb5ce7cba2be6561505633193a"
server_key_version: "encryptionkey@0"
I20260923 14:36:39.122023 8710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "3ef5f42c4ba7428cb4abebdaaf7acc80"
format_stamp: "Formatted at 2026-09-23 14:36:39 on dist-test-slave-zl47"
server_key: "67b92c59f38c8baa781c3d8e91d173c2"
server_key_iv: "763b6ceb5ce7cba2be6561505633193a"
server_key_version: "encryptionkey@0"
I20260923 14:36:39.125618 8710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:36:39.127626 8725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:39.128603 8710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:39.128682 8710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "3ef5f42c4ba7428cb4abebdaaf7acc80"
format_stamp: "Formatted at 2026-09-23 14:36:39 on dist-test-slave-zl47"
server_key: "67b92c59f38c8baa781c3d8e91d173c2"
server_key_iv: "763b6ceb5ce7cba2be6561505633193a"
server_key_version: "encryptionkey@0"
I20260923 14:36:39.128747 8710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:39.146955 8710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:39.149348 8710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:39.149466 8710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:39.149852 8710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:39.150542 8710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:39.150583 8710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:39.150615 8710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:39.150629 8710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:39.158499 8710 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:45905
I20260923 14:36:39.158527 8838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:45905 every 8 connection(s)
I20260923 14:36:39.159209 8710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:36:39.164386 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 8710
I20260923 14:36:39.164543 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:39.164891 2920 external_mini_cluster.cc:1428] Setting key 4d930673d9a6a180523617a4bbfb59e8
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:39.172374 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:39.160661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:48051 (local address 127.2.218.62:38309)
0923 14:36:39.160781 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:39.160788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:39.161395 (+ 607us) server_negotiation.cc:407] Connection header received
0923 14:36:39.162632 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:39.162635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:39.162692 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:39.162770 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:39.163799 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.164320 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.165727 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.166042 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.166185 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:39.168833 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:39.168838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:39.168846 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:39.168870 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:39.170387 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:39.170884 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:39.170892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:39.170893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:39.170965 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:39.171302 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:39.171305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:39.171307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:39.171473 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:39.171606 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:39.172067 (+ 461us) server_negotiation.cc:299] Negotiation successful
0923 14:36:39.172203 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":45}
I20260923 14:36:39.173086 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:39.160869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38309 (local address 127.2.218.2:48051)
0923 14:36:39.161251 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:39.161288 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:39.162398 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:39.162910 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:39.162916 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:39.163246 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:39.163666 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:39.163679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.164454 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.164460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:39.165605 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:39.165614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.166219 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.166224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:39.166421 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:39.166978 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:39.167003 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:39.168673 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:39.170504 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:39.170509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:39.170521 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:39.170775 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:39.171054 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:39.171058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:39.171060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:39.171198 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:39.171611 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:39.171619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:39.171831 (+ 212us) client_negotiation.cc:769] Sending connection context
0923 14:36:39.172018 (+ 187us) client_negotiation.cc:240] Negotiation successful
0923 14:36:39.172303 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":99,"threads_started":1}
I20260923 14:36:39.174157 8839 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38309
I20260923 14:36:39.174402 8839 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:39.174921 8839 heartbeater.cc:511] Master 127.2.218.62:38309 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
I20260923 14:36:39.175957 8514 ts_manager.cc:194] Registered new tserver with Master: 3ef5f42c4ba7428cb4abebdaaf7acc80 (127.2.218.2:45905)
I20260923 14:36:39.176498 8514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:48051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:36:39.190824 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38309
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:39.289860 8844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:39.290058 8844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:39.290133 8844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:39.293850 8844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:39.293895 8844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:39.293974 8844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:39.298697 8844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:39.299479 8844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:39.300302 8844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:39.308643 8852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:39.308694 8849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:39.308670 8850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:39.308776 8844 server_base.cc:1034] running on GCE node
I20260923 14:36:39.309237 8844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:39.309768 8844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:39.310957 8844 hybrid_clock.cc:648] HybridClock initialized: now 1790174199310938 us; error 38 us; skew 500 ppm
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:39.313316 8844 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:36:39.314006 8844 webserver.cc:460] Webserver started at http://127.2.218.3:39263/ using document root <none> and password file <none>
I20260923 14:36:39.314520 8844 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:39.314579 8844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:39.314769 8844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:39.316463 8844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "40d43dfa993241d9acc37547c3198612"
format_stamp: "Formatted at 2026-09-23 14:36:39 on dist-test-slave-zl47"
server_key: "f8b1a22220d384476decd76e64b8e3b8"
server_key_iv: "26344994a420aaf4dcf73ff79523ddb3"
server_key_version: "encryptionkey@0"
I20260923 14:36:39.316907 8844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "40d43dfa993241d9acc37547c3198612"
format_stamp: "Formatted at 2026-09-23 14:36:39 on dist-test-slave-zl47"
server_key: "f8b1a22220d384476decd76e64b8e3b8"
server_key_iv: "26344994a420aaf4dcf73ff79523ddb3"
server_key_version: "encryptionkey@0"
I20260923 14:36:39.319984 8844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:39.322521 8859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:39.323836 8844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:36:39.323939 8844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "40d43dfa993241d9acc37547c3198612"
format_stamp: "Formatted at 2026-09-23 14:36:39 on dist-test-slave-zl47"
server_key: "f8b1a22220d384476decd76e64b8e3b8"
server_key_iv: "26344994a420aaf4dcf73ff79523ddb3"
server_key_version: "encryptionkey@0"
I20260923 14:36:39.324033 8844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:39.343860 8844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:39.346446 8844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:39.346629 8844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:39.347158 8844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:39.347826 8844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:39.347867 8844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:39.347929 8844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:39.347967 8844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:39.355391 8844 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:42945
I20260923 14:36:39.355396 8972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:42945 every 8 connection(s)
I20260923 14:36:39.356067 8844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:39.365615 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 8844
I20260923 14:36:39.365749 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:39.366015 2920 external_mini_cluster.cc:1428] Setting key d29b88080af9ae6d47c6fd444e92c992
I20260923 14:36:39.368367 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:39.357665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:43149 (local address 127.2.218.62:38309)
0923 14:36:39.357812 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:39.357817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:39.358545 (+ 728us) server_negotiation.cc:407] Connection header received
0923 14:36:39.359675 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:39.359678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:39.359737 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:39.359804 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:39.360713 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.361205 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.362314 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.362526 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.362606 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:39.364776 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:39.364779 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:39.364781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:39.364803 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:39.366294 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:39.366844 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:39.366847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:39.366848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:39.366937 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:39.367317 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:39.367322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:39.367323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:39.367521 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:39.367615 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:39.368088 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 14:36:39.368196 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:36:39.369063 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:39.357975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38309 (local address 127.2.218.3:43149)
0923 14:36:39.358419 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:39.358453 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:39.359497 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:39.359941 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:39.359947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:39.360237 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:39.360596 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:39.360606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.361301 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.361303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:39.362212 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:39.362222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.362662 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.362665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:39.362805 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:39.363272 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:39.363287 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:39.364611 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:39.366412 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:39.366416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:39.366445 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:39.366752 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:39.367030 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:39.367033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:39.367035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:39.367204 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:39.367617 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:39.367621 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:39.367839 (+ 218us) client_negotiation.cc:769] Sending connection context
0923 14:36:39.368048 (+ 209us) client_negotiation.cc:240] Negotiation successful
0923 14:36:39.368335 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":168,"threads_started":1}
I20260923 14:36:39.370028 8973 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38309
I20260923 14:36:39.370257 8973 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:39.370616 8973 heartbeater.cc:511] Master 127.2.218.62:38309 requested a full tablet report, sending...
I20260923 14:36:39.371436 8514 ts_manager.cc:194] Registered new tserver with Master: 40d43dfa993241d9acc37547c3198612 (127.2.218.3:42945)
I20260923 14:36:39.371891 8514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:43149
I20260923 14:36:39.379493 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/test-admin.keytab.
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:39.394191 2920 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 14:36:39.396617 8980 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:39.395311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.2.218.1:34195)
0923 14:36:39.395611 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:39.395619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:39.395640 (+ 21us) server_negotiation.cc:407] Connection header received
0923 14:36:39.395693 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:39.395697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:39.395838 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:39.395962 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:39.396432 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39288: BlockingRecv error: recv got EOF from 127.0.0.1:39288 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":117,"threads_started":1}
W20260923 14:36:39.396684 2920 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5cc' of type FILE
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:39.397625 2920 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:39.398556 2920 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:36:39.407122 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:39.399334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.2.218.1:34195)
0923 14:36:39.399455 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:39.399459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:39.399551 (+ 92us) server_negotiation.cc:407] Connection header received
0923 14:36:39.399707 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:39.399710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:39.399754 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:39.399836 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:39.400336 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.401062 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.401860 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.402238 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.402342 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:39.404244 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:39.404252 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:39.404261 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:39.404281 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:39.405642 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:39.406011 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:39.406015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:39.406016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:39.406082 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:39.406373 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:39.406375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:39.406376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:39.406563 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:39.406719 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:39.406890 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 14:36:39.406998 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 14:36:39.410452 8980 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:39.409467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.2.218.1:34195)
0923 14:36:39.409597 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:39.409600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:39.409742 (+ 142us) server_negotiation.cc:407] Connection header received
0923 14:36:39.409844 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:39.409848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:39.409885 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:39.409965 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:39.410362 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39302: BlockingRecv error: recv got EOF from 127.0.0.1:39302 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 14:36:39.410759 2920 init.cc:244] Error while opening credential cache '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5cc)
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:39.411580 2920 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:39.412352 2920 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:36:39.419466 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:39.413012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39306 (local address 127.2.218.1:34195)
0923 14:36:39.413139 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:39.413142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:39.413154 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:39.413241 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:39.413243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:39.413271 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:39.413327 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:39.413724 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.414219 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.414843 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.415021 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.415102 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:39.416809 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:39.416812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:39.416813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:39.416830 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:39.418186 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:39.418515 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:39.418517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:39.418518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:39.418600 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:39.418812 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:39.418814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:39.418815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:39.418943 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:39.419030 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:39.419195 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 14:36:39.419298 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 14:36:39.421690 8980 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:39.420927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.2.218.1:34195)
0923 14:36:39.421009 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:39.421011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:39.421094 (+ 83us) server_negotiation.cc:407] Connection header received
0923 14:36:39.421166 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:39.421168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:39.421195 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:39.421250 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:39.421565 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39314: BlockingRecv error: recv got EOF from 127.0.0.1:39314 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260923 14:36:39.421748 2920 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790174173335390-2920-0/krb5kdc/krb5cc' of type FILE
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:39.422641 2920 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:39.423431 2920 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:36:39.430785 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:39.424071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.2.218.1:34195)
0923 14:36:39.424180 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:39.424184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:39.424229 (+ 45us) server_negotiation.cc:407] Connection header received
0923 14:36:39.424303 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:39.424305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:39.424331 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:39.424384 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:39.424730 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.425114 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.425731 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:39.425904 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:39.425976 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:39.428037 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:39.428040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:39.428042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:39.428070 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:39.429353 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:39.429754 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:39.429758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:39.429760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:39.429831 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:39.430087 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:39.430112 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:39.430114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:39.430246 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:39.430360 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:39.430571 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 14:36:39.430670 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:36:39.431493 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 8576
I20260923 14:36:39.437811 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 8710
I20260923 14:36:39.442719 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 8844
I20260923 14:36:39.447299 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 8482
2026-09-23T14:36:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1953 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8993](info): set up 2 sockets
Sep 23 14:36:39 dist-test-slave-zl47 krb5kdc[8993](info): commencing operation
krb5kdc: starting...
W20260923 14:36:40.426959 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:40 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.62.keytab.
Entry for principal oryx/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.62.keytab.
I20260923 14:36:40.541049 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:39787
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.62.keytab
--principal=oryx/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:39787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:40.639123 9006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:40.639370 9006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:40.639439 9006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:40.643214 9006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:40.643281 9006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:40.643316 9006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:40.643368 9006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:40.643401 9006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:40.647953 9006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:39787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.62.keytab
--principal=oryx/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:39787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:40.648773 9006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:40.649600 9006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:36:40.657477 9006 server_base.cc:1034] running on GCE node
W20260923 14:36:40.657449 9011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:40.657428 9012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:40.657409 9014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:40.658262 9006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:40.659164 9006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:40.660320 9006 hybrid_clock.cc:648] HybridClock initialized: now 1790174200660303 us; error 29 us; skew 500 ppm
Sep 23 14:36:40 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174200, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:40.662876 9006 init.cc:377] Logged in from keytab as oryx/127.2.218.62@KRBTEST.COM (short username oryx)
I20260923 14:36:40.663697 9006 webserver.cc:460] Webserver started at http://127.2.218.62:36659/ using document root <none> and password file <none>
I20260923 14:36:40.664216 9006 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:40.664299 9006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:40.664506 9006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:40.666356 9006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "d24f06a6135f49f2bdaf29e7f60bb586"
format_stamp: "Formatted at 2026-09-23 14:36:40 on dist-test-slave-zl47"
server_key: "2961d0326559fd902e3a6335138fb0b0"
server_key_iv: "1405b27c84e5b95cc8bc8da5959ab664"
server_key_version: "encryptionkey@0"
I20260923 14:36:40.666913 9006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "d24f06a6135f49f2bdaf29e7f60bb586"
format_stamp: "Formatted at 2026-09-23 14:36:40 on dist-test-slave-zl47"
server_key: "2961d0326559fd902e3a6335138fb0b0"
server_key_iv: "1405b27c84e5b95cc8bc8da5959ab664"
server_key_version: "encryptionkey@0"
I20260923 14:36:40.670054 9006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:36:40.672217 9021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:40.673197 9006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:40.673318 9006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "d24f06a6135f49f2bdaf29e7f60bb586"
format_stamp: "Formatted at 2026-09-23 14:36:40 on dist-test-slave-zl47"
server_key: "2961d0326559fd902e3a6335138fb0b0"
server_key_iv: "1405b27c84e5b95cc8bc8da5959ab664"
server_key_version: "encryptionkey@0"
I20260923 14:36:40.673410 9006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:40.712776 9006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:40.715744 9006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:40.715935 9006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:40.722129 9006 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:39787
I20260923 14:36:40.722383 9073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:39787 every 8 connection(s)
I20260923 14:36:40.723019 9006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:40.725523 9074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:40.725939 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 9006
I20260923 14:36:40.726081 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:40.726366 2920 external_mini_cluster.cc:1428] Setting key 034bfa184f73d7ba0410491f39a59a9a
I20260923 14:36:40.732421 9074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586: Bootstrap starting.
Sep 23 14:36:40 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:40.735342 9074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:40.736117 9074 log.cc:824] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:40.738068 9074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586: No bootstrap required, opened a new log
I20260923 14:36:40.740319 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:40.727548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.2.218.62:39787)
0923 14:36:40.727916 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:40.727932 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:40.727974 (+ 42us) server_negotiation.cc:407] Connection header received
0923 14:36:40.728564 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:40.728577 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:40.728852 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:40.729086 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:40.730201 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:40.731115 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:40.732394 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:40.732876 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:40.732985 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:40.735889 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:40.735894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:40.735905 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:40.735930 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:40.737579 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:40.738025 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:40.738035 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:40.738039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:40.738221 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:40.738588 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:40.738591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:40.738592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:40.738925 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:40.739061 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:40.739398 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 14:36:40.739659 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":119,"threads_started":1}
I20260923 14:36:40.742552 9074 raft_consensus.cc:348] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } }
I20260923 14:36:40.742799 9074 raft_consensus.cc:374] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:40.742889 9074 raft_consensus.cc:729] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24f06a6135f49f2bdaf29e7f60bb586, State: Initialized, Role: FOLLOWER
I20260923 14:36:40.743350 9074 consensus_queue.cc:260] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } }
I20260923 14:36:40.743520 9074 raft_consensus.cc:388] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:40.743618 9074 raft_consensus.cc:482] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:40.743752 9074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:40.744709 9074 raft_consensus.cc:504] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } }
I20260923 14:36:40.745086 9074 leader_election.cc:302] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d24f06a6135f49f2bdaf29e7f60bb586; no voters:
I20260923 14:36:40.745478 9074 leader_election.cc:288] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:40.745556 9079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:40.745793 9079 raft_consensus.cc:686] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 1 LEADER]: Becoming Leader. State: Replica: d24f06a6135f49f2bdaf29e7f60bb586, State: Running, Role: LEADER
I20260923 14:36:40.746129 9079 consensus_queue.cc:237] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } }
I20260923 14:36:40.746790 9074 sys_catalog.cc:564] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:40.747629 9081 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d24f06a6135f49f2bdaf29e7f60bb586. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } }
I20260923 14:36:40.747756 9081 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:40.747712 9080 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } }
I20260923 14:36:40.747931 9080 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:40.748324 9086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:40.752068 9086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:40.758230 9086 catalog_manager.cc:1347] Generated new cluster ID: 7a542e1571b642c9817a2e9f29ac9915
I20260923 14:36:40.758301 9086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:40.765897 9086 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:40.767172 9086 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:40.782938 9086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586: Generated new TSK 0
I20260923 14:36:40.783643 9086 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab.
Entry for principal oryx/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab.
I20260923 14:36:40.814880 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39787
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab
--principal=oryx/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:40.919960 9100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:40.920200 9100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:40.920274 9100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:40.924094 9100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:40.924173 9100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:40.924257 9100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:40.929075 9100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab
--principal=oryx/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:40.929940 9100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:40.930886 9100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:40.939406 9106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:40.939416 9105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:40.939440 9108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:40.939796 9100 server_base.cc:1034] running on GCE node
I20260923 14:36:40.940126 9100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:40.940658 9100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:40.941811 9100 hybrid_clock.cc:648] HybridClock initialized: now 1790174200941792 us; error 46 us; skew 500 ppm
Sep 23 14:36:40 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174200, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:40.944382 9100 init.cc:377] Logged in from keytab as oryx/127.2.218.1@KRBTEST.COM (short username oryx)
I20260923 14:36:40.945168 9100 webserver.cc:460] Webserver started at http://127.2.218.1:45525/ using document root <none> and password file <none>
I20260923 14:36:40.945654 9100 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:40.945710 9100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:40.945930 9100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:40.948083 9100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "50c6accd5b564cc5a4cccba961f80355"
format_stamp: "Formatted at 2026-09-23 14:36:40 on dist-test-slave-zl47"
server_key: "d11088f5f234b4b32d304412b4104287"
server_key_iv: "50e99622dda2a0e8c2c11fc9267e848e"
server_key_version: "encryptionkey@0"
I20260923 14:36:40.948632 9100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "50c6accd5b564cc5a4cccba961f80355"
format_stamp: "Formatted at 2026-09-23 14:36:40 on dist-test-slave-zl47"
server_key: "d11088f5f234b4b32d304412b4104287"
server_key_iv: "50e99622dda2a0e8c2c11fc9267e848e"
server_key_version: "encryptionkey@0"
I20260923 14:36:40.952198 9100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:36:40.954389 9115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:40.955488 9100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260923 14:36:40.955623 9100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "50c6accd5b564cc5a4cccba961f80355"
format_stamp: "Formatted at 2026-09-23 14:36:40 on dist-test-slave-zl47"
server_key: "d11088f5f234b4b32d304412b4104287"
server_key_iv: "50e99622dda2a0e8c2c11fc9267e848e"
server_key_version: "encryptionkey@0"
I20260923 14:36:40.955720 9100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:40.969825 9100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:40.972340 9100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:40.972554 9100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:40.973043 9100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:40.973867 9100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:40.973917 9100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:40.973984 9100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:40.974021 9100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:40.982969 9100 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:39005
I20260923 14:36:40.982990 9228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:39005 every 8 connection(s)
I20260923 14:36:40.983757 9100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:40.990301 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 9100
I20260923 14:36:40.990437 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:40.990731 2920 external_mini_cluster.cc:1428] Setting key fb3aa2dfd81e9e99071a6e389e3a68ad
Sep 23 14:36:40 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174200, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:40.997079 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:40.985307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:38899 (local address 127.2.218.62:39787)
0923 14:36:40.985456 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:40.985462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:40.986273 (+ 811us) server_negotiation.cc:407] Connection header received
0923 14:36:40.987417 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:40.987420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:40.987470 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:40.987551 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:40.988788 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:40.989352 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:40.990845 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:40.991101 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:40.991193 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:40.993539 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:40.993542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:40.993544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:40.993566 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:40.995021 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:40.995528 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:40.995534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:40.995539 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:40.995628 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:40.995934 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:40.995940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:40.995945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:40.996191 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:40.996301 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:40.996779 (+ 478us) server_negotiation.cc:299] Negotiation successful
0923 14:36:40.996905 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:36:40.997610 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:40.985671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.1:38899)
0923 14:36:40.986064 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:40.986153 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:40.987204 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:40.987701 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:40.987707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:40.988236 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:40.988644 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:40.988656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:40.989465 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:40.989467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:40.990698 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:40.990708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:40.991253 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:40.991261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:40.991448 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:40.992029 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:40.992044 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:40.993316 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:40.995187 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:40.995191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:40.995199 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:40.995429 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:40.995727 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:40.995729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:40.995730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:40.995844 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:40.996295 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:40.996299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:40.996536 (+ 237us) client_negotiation.cc:769] Sending connection context
0923 14:36:40.996703 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 14:36:40.996913 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
I20260923 14:36:40.998836 9229 heartbeater.cc:347] Connected to a master server at 127.2.218.62:39787
I20260923 14:36:40.999049 9229 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:40.999612 9229 heartbeater.cc:511] Master 127.2.218.62:39787 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.2.218.2@KRBTEST.COM; defaulting to no policy
I20260923 14:36:41.001255 9038 ts_manager.cc:194] Registered new tserver with Master: 50c6accd5b564cc5a4cccba961f80355 (127.2.218.1:39005)
I20260923 14:36:41.002406 9038 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:38899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab.
Entry for principal oryx/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab.
I20260923 14:36:41.016695 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39787
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab
--principal=oryx/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:41.113547 9234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:41.113766 9234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:41.113834 9234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:41.117655 9234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:41.117733 9234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:41.117817 9234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:41.123211 9234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab
--principal=oryx/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:41.124022 9234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:41.124858 9234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:41.133514 9239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:41.133539 9242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:41.133644 9234 server_base.cc:1034] running on GCE node
W20260923 14:36:41.133569 9240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:41.134057 9234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:41.134625 9234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:41.135807 9234 hybrid_clock.cc:648] HybridClock initialized: now 1790174201135792 us; error 32 us; skew 500 ppm
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:41.138342 9234 init.cc:377] Logged in from keytab as oryx/127.2.218.2@KRBTEST.COM (short username oryx)
I20260923 14:36:41.139092 9234 webserver.cc:460] Webserver started at http://127.2.218.2:34673/ using document root <none> and password file <none>
I20260923 14:36:41.139613 9234 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:41.139671 9234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:41.139896 9234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:41.141770 9234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "b476e81a19fc420bb11e0666879be1e7"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "67e9894275282f5268de44097f8d7431"
server_key_iv: "575457b68116c0fd9a6bdb8f45dc2134"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.142292 9234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "b476e81a19fc420bb11e0666879be1e7"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "67e9894275282f5268de44097f8d7431"
server_key_iv: "575457b68116c0fd9a6bdb8f45dc2134"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.145462 9234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:36:41.147696 9249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.148837 9234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:41.148959 9234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "b476e81a19fc420bb11e0666879be1e7"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "67e9894275282f5268de44097f8d7431"
server_key_iv: "575457b68116c0fd9a6bdb8f45dc2134"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.149101 9234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:41.175199 9234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:41.177922 9234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:41.178174 9234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:41.178676 9234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:41.179453 9234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:41.179523 9234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.179589 9234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:41.179644 9234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.187768 9234 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:46199
I20260923 14:36:41.187798 9362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:46199 every 8 connection(s)
I20260923 14:36:41.188511 9234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:36:41.191848 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 9234
I20260923 14:36:41.191979 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:41.192263 2920 external_mini_cluster.cc:1428] Setting key 4dc3a3685f02057842f46e2355a75e1b
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.2.218.3@KRBTEST.COM; defaulting to no policy
I20260923 14:36:41.201406 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.190161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:45327 (local address 127.2.218.62:39787)
0923 14:36:41.190288 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.190298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.191024 (+ 726us) server_negotiation.cc:407] Connection header received
0923 14:36:41.192371 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.192375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.192432 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.192508 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.193661 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.194177 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.195181 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.195403 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.195505 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.197914 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.197921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.197926 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.197969 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.199448 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.199893 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.199898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.199900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.199979 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.200273 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.200276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.200278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.200532 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.200656 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.201080 (+ 424us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.201195 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 14:36:41.202028 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.190501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.2:45327)
0923 14:36:41.190889 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.190924 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.192166 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.192668 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.192674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.193021 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.193519 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.193532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.194288 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.194293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.195059 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.195067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.195535 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.195539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.195690 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.196182 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.196196 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.197756 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.199564 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.199568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.199577 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.199799 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.200068 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.200070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.200071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.200186 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.200649 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.200654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.200855 (+ 201us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.200996 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.201223 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
I20260923 14:36:41.203266 9363 heartbeater.cc:347] Connected to a master server at 127.2.218.62:39787
I20260923 14:36:41.203444 9363 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.3@KRBTEST.COM" created.
I20260923 14:36:41.203922 9363 heartbeater.cc:511] Master 127.2.218.62:39787 requested a full tablet report, sending...
I20260923 14:36:41.204865 9038 ts_manager.cc:194] Registered new tserver with Master: b476e81a19fc420bb11e0666879be1e7 (127.2.218.2:46199)
I20260923 14:36:41.205375 9038 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:45327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab.
Entry for principal oryx/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab.
I20260923 14:36:41.218269 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39787
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab
--principal=oryx/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:41.314553 9368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:41.314754 9368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:41.314791 9368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:41.318259 9368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:41.318295 9368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:41.318342 9368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:41.322842 9368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab
--principal=oryx/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:41.323557 9368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:41.324294 9368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:41.332564 9376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:41.332558 9373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:41.332559 9374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:41.332746 9368 server_base.cc:1034] running on GCE node
I20260923 14:36:41.333148 9368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:41.333711 9368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:41.334882 9368 hybrid_clock.cc:648] HybridClock initialized: now 1790174201334869 us; error 30 us; skew 500 ppm
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:41.337260 9368 init.cc:377] Logged in from keytab as oryx/127.2.218.3@KRBTEST.COM (short username oryx)
I20260923 14:36:41.338050 9368 webserver.cc:460] Webserver started at http://127.2.218.3:44759/ using document root <none> and password file <none>
I20260923 14:36:41.338542 9368 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:41.338624 9368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:41.338833 9368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:41.340508 9368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "c68f17b782494aed8dcec316efb7a76f"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "fb90aa5d5b9d38675651632740c75b18"
server_key_iv: "2c27ee5d0bc933068837ff05e17112c0"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.341012 9368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "c68f17b782494aed8dcec316efb7a76f"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "fb90aa5d5b9d38675651632740c75b18"
server_key_iv: "2c27ee5d0bc933068837ff05e17112c0"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.344198 9368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:41.346333 9383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.347445 9368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:41.347577 9368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "c68f17b782494aed8dcec316efb7a76f"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "fb90aa5d5b9d38675651632740c75b18"
server_key_iv: "2c27ee5d0bc933068837ff05e17112c0"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.347710 9368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:41.361979 9368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:41.364374 9368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:41.364539 9368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:41.365015 9368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:41.365693 9368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:41.365758 9368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.365816 9368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:41.365859 9368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.373965 9368 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:37507
I20260923 14:36:41.374009 9496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:37507 every 8 connection(s)
I20260923 14:36:41.374717 9368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:41.382972 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 9368
I20260923 14:36:41.383121 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:41.383405 2920 external_mini_cluster.cc:1428] Setting key d1ba807771b7124d7c7b490d6aed7132
I20260923 14:36:41.387149 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.376165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:52995 (local address 127.2.218.62:39787)
0923 14:36:41.376295 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.376300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.377023 (+ 723us) server_negotiation.cc:407] Connection header received
0923 14:36:41.378292 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.378295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.378338 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.378407 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.379405 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.379921 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.380956 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.381164 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.381237 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.383600 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.383605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.383607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.383634 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.385201 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.385668 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.385671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.385675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.385746 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.386057 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.386060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.386062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.386275 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.386374 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.386873 (+ 499us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.386980 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:36:41.387763 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.376478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.3:52995)
0923 14:36:41.376898 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.376935 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.378060 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.378551 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.378557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.378869 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.379276 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.379288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.380035 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.380037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.380853 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.380862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.381276 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.381279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.381400 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.381911 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.381925 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.383387 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.385307 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.385312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.385323 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.385550 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.385838 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.385840 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.385842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.385966 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.386373 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.386378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.386582 (+ 204us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.386810 (+ 228us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.387086 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":109,"threads_started":1}
I20260923 14:36:41.388732 9497 heartbeater.cc:347] Connected to a master server at 127.2.218.62:39787
I20260923 14:36:41.388931 9497 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:41.389335 9497 heartbeater.cc:511] Master 127.2.218.62:39787 requested a full tablet report, sending...
I20260923 14:36:41.390245 9038 ts_manager.cc:194] Registered new tserver with Master: c68f17b782494aed8dcec316efb7a76f (127.2.218.3:37507)
I20260923 14:36:41.390852 9038 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:52995
I20260923 14:36:41.397076 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.218.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.61.keytab.
Entry for principal oryx/127.2.218.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.61.keytab.
I20260923 14:36:41.421604 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.61:45373
--webserver_interface=127.2.218.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.61.keytab
--principal=oryx/127.2.218.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:39787,127.2.218.61:45373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:41.517107 9502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:41.517293 9502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:41.517328 9502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:41.520538 9502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:41.520581 9502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:41.520601 9502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:41.520617 9502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:41.520632 9502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:41.524744 9502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:39787,127.2.218.61:45373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.61.keytab
--principal=oryx/127.2.218.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.61:45373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.218.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
W20260923 14:36:41.524874 9502 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.218.62:39787,127.2.218.61:45373'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 14:36:41.525446 9502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:41.526180 9502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:36:41.532862 9502 server_base.cc:1034] running on GCE node
W20260923 14:36:41.532821 9510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:41.532816 9507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:41.532825 9508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:41.533387 9502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:41.534245 9502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:41.535385 9502 hybrid_clock.cc:648] HybridClock initialized: now 1790174201535369 us; error 26 us; skew 500 ppm
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:41.537518 9502 init.cc:377] Logged in from keytab as oryx/127.2.218.61@KRBTEST.COM (short username oryx)
I20260923 14:36:41.538233 9502 webserver.cc:460] Webserver started at http://127.2.218.61:32987/ using document root <none> and password file <none>
I20260923 14:36:41.538666 9502 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:41.538738 9502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:41.538918 9502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:41.540518 9502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data/instance:
uuid: "b007bc31c2df4885ada07189c37f05ce"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "973a8cc87c1cf7d885d76e85f8fc9cd1"
server_key_iv: "0bf2944842b3c81f4536866b7ef8506c"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.541033 9502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal/instance:
uuid: "b007bc31c2df4885ada07189c37f05ce"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "973a8cc87c1cf7d885d76e85f8fc9cd1"
server_key_iv: "0bf2944842b3c81f4536866b7ef8506c"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.543960 9502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:41.545825 9517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.546815 9502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:41.546929 9502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal
uuid: "b007bc31c2df4885ada07189c37f05ce"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "973a8cc87c1cf7d885d76e85f8fc9cd1"
server_key_iv: "0bf2944842b3c81f4536866b7ef8506c"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.547055 9502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:41.560088 9502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:41.562242 9502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:41.562394 9502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:41.567848 9502 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.61:45373
I20260923 14:36:41.568097 9569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.61:45373 every 8 connection(s)
I20260923 14:36:41.568580 9502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
I20260923 14:36:41.570763 9570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:41.573629 9570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:36:41.576159 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 9502
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.61@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.218.61@KRBTEST.COM
I20260923 14:36:41.586074 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.575127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.2.218.62:39787)
0923 14:36:41.575239 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.575243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.575873 (+ 630us) server_negotiation.cc:407] Connection header received
0923 14:36:41.576692 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.576695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.576783 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.576853 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.578064 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.578813 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.579841 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.580022 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.580132 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.582434 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.582439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.582446 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.582467 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.583937 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.584646 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.584650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.584652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.584758 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.585089 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.585094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.585096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.585308 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.585422 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.585837 (+ 415us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.585934 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:36:41.586658 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.576858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.2.218.61:45373)
0923 14:36:41.577180 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.577187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.577204 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:36:41.577329 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.577334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.577467 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.577577 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.578356 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.579257 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.580113 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.580452 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.580549 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.582859 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.582867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.582877 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.582898 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.584669 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.585100 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.585105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.585107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.585195 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.585486 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.585490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.585492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.585690 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.585853 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.586053 (+ 200us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.586387 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260923 14:36:41.586660 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.575377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:34676)
0923 14:36:41.575763 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.575795 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.576480 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.576992 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.577000 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.577425 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.577914 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.577930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.578929 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.578932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.579734 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.579745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.580163 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.580166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.580282 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.580777 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.580799 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.582306 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.584217 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.584224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.584230 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.584526 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.584858 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.584861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.584863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.584990 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.585443 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.585450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.585632 (+ 182us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.585788 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.586017 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
I20260923 14:36:41.587798 9570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.61@KRBTEST.COM for oryx/127.2.218.61@KRBTEST.COM
I20260923 14:36:41.593737 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.588123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:45373 (local address 127.0.0.1:48708)
0923 14:36:41.588230 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.588243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.588297 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.588635 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.588637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.588739 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.588809 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.588812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.589356 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.589358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.589765 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.589771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.590144 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.590148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.590216 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.590576 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.590587 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.591596 (+ 1009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.592757 (+ 1161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.592759 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.592760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.592912 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.593107 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.593108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.593109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.593173 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.593427 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.593429 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.593483 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.593547 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.593618 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 14:36:41.593765 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.588395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.2.218.61:45373)
0923 14:36:41.588493 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.588495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.588504 (+ 9us) server_negotiation.cc:407] Connection header received
0923 14:36:41.588527 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.588528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.588553 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.588606 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.588888 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.589269 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.589845 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.590024 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.590084 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.591695 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.591697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.591699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.591715 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.592672 (+ 957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.592983 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.592985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.592986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.593041 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.593246 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.593248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.593250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.593362 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.593421 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.593592 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.593658 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:36:41.595988 9570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: Bootstrap starting.
I20260923 14:36:41.597710 9570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:41.598356 9570 log.cc:824] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:41.599696 9570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: No bootstrap required, opened a new log
I20260923 14:36:41.601976 9570 raft_consensus.cc:348] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } }
I20260923 14:36:41.602180 9570 raft_consensus.cc:374] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:41.602233 9570 raft_consensus.cc:729] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b007bc31c2df4885ada07189c37f05ce, State: Initialized, Role: FOLLOWER
I20260923 14:36:41.602574 9570 consensus_queue.cc:260] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } }
I20260923 14:36:41.602922 9576 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } } }
I20260923 14:36:41.603011 9576 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:36:41.603461 9570 sys_catalog.cc:564] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: configured and running, proceeding with master startup.
W20260923 14:36:41.607229 9587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:36:41.607299 9587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:36:41.608484 9502 master_runner.cc:428] Detected that this master b007bc31c2df4885ada07189c37f05ce is joining an existing cluster
I20260923 14:36:41.608544 9502 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.218.61:45373
I20260923 14:36:41.609663 9576 raft_consensus.cc:482] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:41.609784 9576 raft_consensus.cc:504] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } }
I20260923 14:36:41.610467 9576 leader_election.cc:288] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787)
I20260923 14:36:41.610946 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b007bc31c2df4885ada07189c37f05ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" is_pre_election: true
I20260923 14:36:41.611474 9518 leader_election.cc:302] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b007bc31c2df4885ada07189c37f05ce; no voters: d24f06a6135f49f2bdaf29e7f60bb586
I20260923 14:36:41.611691 9576 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 14:36:41.621001 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.610079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.2.218.62:39787)
0923 14:36:41.610229 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.610233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.614332 (+ 4099us) server_negotiation.cc:407] Connection header received
0923 14:36:41.614422 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.614426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.614465 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.614530 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.614973 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.615465 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.616948 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.617236 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.617321 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.618197 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.618200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.618202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.618224 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.619583 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.620026 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.620029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.620030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.620101 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.620333 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.620336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.620337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.620482 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.620582 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.620789 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.620889 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 14:36:41.621021 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.610021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:34680)
0923 14:36:41.614259 (+ 4238us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.614268 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.614360 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.614572 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.614574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.614750 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.614865 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.614871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.615558 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.615562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.616838 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.616847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.617343 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.617345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.617400 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.617796 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.617814 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.618046 (+ 232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.619680 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.619683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.619685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.619937 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.620182 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.620185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.620187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.620265 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.620575 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.620578 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.620648 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.620752 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.620874 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4174,"mutex_wait_us":35,"spinlock_wait_cycles":2944,"thread_start_us":170,"threads_started":1}
I20260923 14:36:41.622300 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.610476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:45373 (local address 127.0.0.1:48718)
0923 14:36:41.614538 (+ 4062us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.614548 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.614617 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.614794 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.614796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.614959 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.615052 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.615057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.615880 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.615883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.616838 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.616847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.617877 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.617880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.617951 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.618341 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.618350 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.618680 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.620272 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.620275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.620277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.620480 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.621440 (+ 960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.621443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.621444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.621524 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.621893 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.621895 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.621952 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.622032 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.622152 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4004,"mutex_wait_us":118,"thread_start_us":120,"threads_started":1}
I20260923 14:36:41.622328 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.610562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48718 (local address 127.2.218.61:45373)
0923 14:36:41.610646 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.610649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.614675 (+ 4026us) server_negotiation.cc:407] Connection header received
0923 14:36:41.614692 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.614693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.614713 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.614783 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.615154 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.615771 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.617406 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.617687 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.617779 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.618805 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.618807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.618809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.618827 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.620182 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.621124 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.621127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.621129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.621193 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.621616 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.621619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.621620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.621787 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.621865 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.622117 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.622202 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28,"spinlock_wait_cycles":768}
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174200, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.62@KRBTEST.COM for oryx/127.2.218.61@KRBTEST.COM
I20260923 14:36:41.631369 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.624406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.2.218.61:45373)
0923 14:36:41.624477 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.624481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.624714 (+ 233us) server_negotiation.cc:407] Connection header received
0923 14:36:41.624880 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.624882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.624918 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.625001 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.625480 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.625860 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.626638 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.626833 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.626926 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.628775 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.628777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.628779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.628804 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.629925 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.630380 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.630384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.630385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.630453 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.630686 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.630690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.630692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.630828 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.630915 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.631174 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.631259 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 14:36:41.631410 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.624382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:45373 (local address 127.0.0.1:48720)
0923 14:36:41.624619 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.624649 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.624768 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.625029 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.625033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.625306 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.625381 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.625386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.625953 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.625956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.626533 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.626541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.626931 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.626934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.627025 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.627430 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.627447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.628657 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.630023 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.630032 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.630036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.630292 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.630530 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.630534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.630536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.630610 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.630919 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.630925 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.631017 (+ 92us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.631132 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.631280 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260923 14:36:41.632411 9038 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.218.61:45373: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b007bc31c2df4885ada07189c37f05ce"
member_type: NON_VOTER
last_known_addr {
host: "127.2.218.61"
port: 45373
}
attrs {
promote: true
}
}
dest_uuid: "d24f06a6135f49f2bdaf29e7f60bb586"
cas_config_opid_index: -1
I20260923 14:36:41.632860 9038 consensus_queue.cc:237] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } }
I20260923 14:36:41.634128 9597 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b007bc31c2df4885ada07189c37f05ce (127.2.218.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } } }
I20260923 14:36:41.634732 9544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:41.635357 9597 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.218.61:45373
I20260923 14:36:41.635654 9502 master.cc:537] Master@127.2.218.61:45373 shutting down...
I20260923 14:36:41.635751 9544 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Refusing update from remote peer d24f06a6135f49f2bdaf29e7f60bb586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 14:36:41.636240 9597 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 14:36:41.637740 9025 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 14:36:41.638505 9025 consensus_peers.cc:487] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 -> Peer b007bc31c2df4885ada07189c37f05ce (127.2.218.61:45373): Couldn't send request to peer b007bc31c2df4885ada07189c37f05ce. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 14:36:41.640683 9596 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } } }
I20260923 14:36:41.640779 9596 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:41.640925 9502 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:41.641119 9502 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:41.641197 9502 tablet_replica.cc:330] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: stopping tablet replica
I20260923 14:36:41.645057 9502 master.cc:552] Master@127.2.218.61:45373 shutdown complete.
I20260923 14:36:41.645098 9502 master_runner.cc:305] Clearing existing system tablet
I20260923 14:36:41.645277 9502 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:41.645323 9502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:41.646381 9502 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 14:36:41.647042 9601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.647219 9502 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:41.647372 9502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:41.647404 9502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal
uuid: "b007bc31c2df4885ada07189c37f05ce"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "973a8cc87c1cf7d885d76e85f8fc9cd1"
server_key_iv: "0bf2944842b3c81f4536866b7ef8506c"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.648854 9502 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:41.650592 9502 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 14:36:41.650676 9502 log.cc:1196] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 14:36:41.650862 9502 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: Deleting consensus metadata
I20260923 14:36:41.651042 9502 master_runner.cc:315] Copying system tablet from 127.2.218.62:39787
I20260923 14:36:41.651842 9502 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: tablet copy: Beginning tablet copy session from remote peer at address 127.2.218.62:39787
I20260923 14:36:41.658294 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.652217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:34686)
0923 14:36:41.652463 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.652472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.652568 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.652755 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.652757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.652938 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.653020 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.653029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.653665 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.653669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.654202 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.654209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.654595 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.654598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.654670 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.655073 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.655106 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.655498 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.657022 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.657027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.657029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.657191 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.657441 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.657445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.657446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.657520 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.657826 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.657829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.657912 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.658001 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.658131 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":73,"threads_started":1}
I20260923 14:36:41.658360 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.652281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.2.218.62:39787)
0923 14:36:41.652368 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.652371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.652536 (+ 165us) server_negotiation.cc:407] Connection header received
0923 14:36:41.652630 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.652632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.652663 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.652736 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.653116 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.653571 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.654309 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.654502 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.654594 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.655616 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.655619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.655622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.655643 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.656914 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.657292 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.657295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.657296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.657362 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.657596 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.657600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.657602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.657735 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.657822 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.658048 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.658150 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:36:41.658826 9058 tablet_copy_service.cc:140] P d24f06a6135f49f2bdaf29e7f60bb586: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b007bc31c2df4885ada07189c37f05ce ({username='oryx', principal='oryx/127.2.218.61@KRBTEST.COM'} at 127.0.0.1:34686)
I20260923 14:36:41.658936 9058 tablet_copy_service.cc:161] P d24f06a6135f49f2bdaf29e7f60bb586: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b007bc31c2df4885ada07189c37f05ce at {username='oryx', principal='oryx/127.2.218.61@KRBTEST.COM'} at 127.0.0.1:34686: session id = b007bc31c2df4885ada07189c37f05ce-00000000000000000000000000000000
I20260923 14:36:41.660003 9058 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 14:36:41.661104 9502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 14:36:41.662436 9502 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: tablet copy: Starting download of 0 data blocks...
I20260923 14:36:41.662590 9502 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: tablet copy: Starting download of 1 WAL segments...
I20260923 14:36:41.663826 9502 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 14:36:41.664953 9058 tablet_copy_service.cc:342] P d24f06a6135f49f2bdaf29e7f60bb586: Request end of tablet copy session b007bc31c2df4885ada07189c37f05ce-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.218.61@KRBTEST.COM'} at 127.0.0.1:34686
I20260923 14:36:41.665019 9058 tablet_copy_service.cc:433] P d24f06a6135f49f2bdaf29e7f60bb586: ending tablet copy session b007bc31c2df4885ada07189c37f05ce-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b007bc31c2df4885ada07189c37f05ce
I20260923 14:36:41.670015 9502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:41.671324 9608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:41.671463 9611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:41.671463 9609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:41.671617 9502 server_base.cc:1034] running on GCE node
I20260923 14:36:41.671770 9502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:41.672057 9502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:41.673182 9502 hybrid_clock.cc:648] HybridClock initialized: now 1790174201673174 us; error 21 us; skew 500 ppm
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:41.674382 9502 init.cc:377] Logged in from keytab as oryx/127.2.218.61@KRBTEST.COM (short username oryx)
I20260923 14:36:41.674738 9502 webserver.cc:460] Webserver started at http://127.2.218.61:45743/ using document root <none> and password file <none>
I20260923 14:36:41.674883 9502 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:41.674917 9502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:41.675853 9502 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:41.676636 9618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.676852 9502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:41.676913 9502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal
uuid: "b007bc31c2df4885ada07189c37f05ce"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "973a8cc87c1cf7d885d76e85f8fc9cd1"
server_key_iv: "0bf2944842b3c81f4536866b7ef8506c"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.676966 9502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:41.682448 9502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:41.684031 9502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:41.687631 9502 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.61:45373
I20260923 14:36:41.687865 9670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.61:45373 every 8 connection(s)
I20260923 14:36:41.688146 9502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
I20260923 14:36:41.689361 9671 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:36:41.690070 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: Bootstrap starting.
I20260923 14:36:41.697454 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:36:41.697715 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: Bootstrap complete.
I20260923 14:36:41.698132 9671 raft_consensus.cc:348] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } }
I20260923 14:36:41.698271 9671 raft_consensus.cc:729] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b007bc31c2df4885ada07189c37f05ce, State: Initialized, Role: LEARNER
I20260923 14:36:41.698374 9671 consensus_queue.cc:260] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } }
I20260923 14:36:41.698930 9675 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } } }
I20260923 14:36:41.699064 9671 sys_catalog.cc:564] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:41.699082 9675 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: This master's current role is: LEARNER
I20260923 14:36:41.702266 9686 catalog_manager.cc:1259] Loaded cluster ID: 7a542e1571b642c9817a2e9f29ac9915
I20260923 14:36:41.702309 9686 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: loading cluster ID for follower catalog manager: success
I20260923 14:36:41.702975 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.694643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.2.218.61:45373)
0923 14:36:41.694861 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.694870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.694884 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:36:41.694916 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.694918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.694953 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.695030 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.695745 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.696715 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.697825 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.698145 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.698249 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.699685 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.699689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.699692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.699715 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.701339 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.701799 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.701803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.701806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.701887 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.702192 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.702195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.702196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.702382 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.702496 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.702731 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.702828 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260923 14:36:41.704587 9686 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: acquiring CA information for follower catalog manager: success
I20260923 14:36:41.705783 9686 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for oryx/127.2.218.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.60.keytab.
Entry for principal oryx/127.2.218.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.60.keytab.
I20260923 14:36:41.725062 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.60:33311
--webserver_interface=127.2.218.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.60.keytab
--principal=oryx/127.2.218.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:39787,127.2.218.61:45373,127.2.218.60:33311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:41.826454 9689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:41.826632 9689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:41.826664 9689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:41.829928 9689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:41.829968 9689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:41.829985 9689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:41.829998 9689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:41.830011 9689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:41.834213 9689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:39787,127.2.218.61:45373,127.2.218.60:33311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.60.keytab
--principal=oryx/127.2.218.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.60:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.218.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:41.834949 9689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:41.835695 9689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:41.842576 9697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:41.842584 9694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:41.842583 9695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:41.842630 9689 server_base.cc:1034] running on GCE node
I20260923 14:36:41.843197 9689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:41.844045 9689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:41.845203 9689 hybrid_clock.cc:648] HybridClock initialized: now 1790174201845184 us; error 35 us; skew 500 ppm
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:41.847991 9689 init.cc:377] Logged in from keytab as oryx/127.2.218.60@KRBTEST.COM (short username oryx)
I20260923 14:36:41.848804 9689 webserver.cc:460] Webserver started at http://127.2.218.60:32899/ using document root <none> and password file <none>
I20260923 14:36:41.849289 9689 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:41.849375 9689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:41.849581 9689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:41.851275 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data/instance:
uuid: "458bac8acab4452e921aa9185b006045"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "a5259565ff705d05651bf047b8df3d82"
server_key_iv: "3cc250befb9ef2857ffc165ab63a2555"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.851732 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal/instance:
uuid: "458bac8acab4452e921aa9185b006045"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "a5259565ff705d05651bf047b8df3d82"
server_key_iv: "3cc250befb9ef2857ffc165ab63a2555"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.854736 9689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:36:41.856603 9704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.857511 9689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:41.857632 9689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal
uuid: "458bac8acab4452e921aa9185b006045"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "a5259565ff705d05651bf047b8df3d82"
server_key_iv: "3cc250befb9ef2857ffc165ab63a2555"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.857720 9689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:41.871564 9689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:41.874218 9689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:41.874393 9689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:41.880647 9689 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.60:33311
I20260923 14:36:41.880898 9756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.60:33311 every 8 connection(s)
I20260923 14:36:41.881453 9689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
I20260923 14:36:41.883404 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 9689
I20260923 14:36:41.884208 9757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:41.887491 9757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.218.60@KRBTEST.COM
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.60@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:41.895494 9759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.884751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.2.218.60:33311)
0923 14:36:41.885107 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.885119 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.885149 (+ 30us) server_negotiation.cc:407] Connection header received
0923 14:36:41.885674 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.885686 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.885924 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.886206 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.886868 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.887665 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.888400 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.888781 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.888926 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.891192 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.891198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.891207 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.891231 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.892859 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.893294 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.893300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.893305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.893436 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.893672 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.893677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.893678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.894120 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.894286 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.894558 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.894805 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
I20260923 14:36:41.897059 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.888879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:34688)
0923 14:36:41.889312 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.889351 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.889468 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.889729 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.889733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.889996 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.890163 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.890170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.890874 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.890877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.891653 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.891668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.892071 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.892077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.892190 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.892583 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.892593 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.894046 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.895691 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.895697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.895700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.895871 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.896116 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.896119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.896121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.896218 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.896556 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.896560 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.896652 (+ 92us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.896767 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.896900 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":123,"threads_started":1}
I20260923 14:36:41.897089 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.888993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.2.218.62:39787)
0923 14:36:41.889122 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.889126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.889436 (+ 310us) server_negotiation.cc:407] Connection header received
0923 14:36:41.889556 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.889559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.889610 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.889686 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.890281 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.890730 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.891780 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.891970 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.892047 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.894221 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.894224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.894226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.894258 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.895568 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.895962 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.895966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.895967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.896039 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.896303 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.896305 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.896306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.896465 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.896543 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.896817 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.896926 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 14:36:41.897908 9757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.60@KRBTEST.COM for oryx/127.2.218.61@KRBTEST.COM
I20260923 14:36:41.904328 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.898288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:45373 (local address 127.0.0.1:48736)
0923 14:36:41.898515 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.898526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.898589 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.898784 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.898785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.898893 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.898965 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.898969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.899601 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.899604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.900073 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.900080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.900530 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.900533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.900600 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.901041 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.901052 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.902093 (+ 1041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.903324 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.903326 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.903327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.903490 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.903695 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.903698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.903700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.903766 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.904026 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.904027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.904088 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.904164 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.904244 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260923 14:36:41.904398 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.898412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48736 (local address 127.2.218.61:45373)
0923 14:36:41.898526 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.898529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.898581 (+ 52us) server_negotiation.cc:407] Connection header received
0923 14:36:41.898664 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.898666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.898699 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.898763 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.899050 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.899512 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.900175 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.900447 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.900514 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.902233 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.902235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.902237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.902255 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.903236 (+ 981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.903565 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.903567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.903568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.903627 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.903833 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.903835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.903836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.903957 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.904018 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.904207 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.904282 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:36:41.904989 9757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174201, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.60@KRBTEST.COM for oryx/127.2.218.60@KRBTEST.COM
I20260923 14:36:41.911360 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.905319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:33311 (local address 127.0.0.1:40006)
0923 14:36:41.905428 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.905438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.905493 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.905631 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.905633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.905740 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.905799 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.905803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.906737 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.906739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.907222 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.907229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.907623 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.907625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.907679 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.908107 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.908119 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.909072 (+ 953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.910273 (+ 1201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.910275 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.910276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.910441 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.910671 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.910673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.910674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.910768 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.911053 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.911055 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.911115 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.911184 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.911259 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 14:36:41.911434 9759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.905410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.2.218.60:33311)
0923 14:36:41.905504 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.905506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.905517 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:36:41.905542 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.905544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.905570 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.905622 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.905881 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.906625 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.907336 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.907528 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.907596 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.909180 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.909182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.909184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.909199 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.910184 (+ 985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.910514 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.910516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.910517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.910572 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.910846 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.910848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.910849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.910971 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.911043 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.911230 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.911296 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:36:41.913329 9757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Bootstrap starting.
I20260923 14:36:41.915082 9757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:41.915658 9757 log.cc:824] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:41.916996 9757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: No bootstrap required, opened a new log
I20260923 14:36:41.919313 9757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } }
I20260923 14:36:41.919479 9757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:41.919554 9757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458bac8acab4452e921aa9185b006045, State: Initialized, Role: FOLLOWER
I20260923 14:36:41.919895 9757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } }
I20260923 14:36:41.920233 9762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } } }
I20260923 14:36:41.920323 9762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:36:41.920763 9757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 14:36:41.924584 9773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:36:41.924652 9773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:36:41.927060 9689 master_runner.cc:428] Detected that this master 458bac8acab4452e921aa9185b006045 is joining an existing cluster
I20260923 14:36:41.927119 9689 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.218.60:33311
I20260923 14:36:41.936626 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.928700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:34700)
0923 14:36:41.929176 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.929185 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.929283 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.929561 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.929563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.929729 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.929845 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.929851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.930752 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.930755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.931493 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.931501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.932079 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.932082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.932175 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.932551 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.932563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.932901 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.934540 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.934543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.934544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.935497 (+ 953us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.935762 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.935765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.935766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.935852 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.936203 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.936206 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.936287 (+ 81us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.936379 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.936484 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260923 14:36:41.936676 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.928999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.2.218.62:39787)
0923 14:36:41.929084 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.929087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.929264 (+ 177us) server_negotiation.cc:407] Connection header received
0923 14:36:41.929356 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.929359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.929397 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.929457 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.929953 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.930640 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.931609 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.931886 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.931943 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.933021 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.933024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.933026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.933046 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.934428 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.935600 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.935603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.935605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.935676 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.935936 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.935939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.935940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.936111 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.936202 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.936446 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.936550 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 14:36:41.936743 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.928756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.2.218.61:45373)
0923 14:36:41.928857 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.928861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.929525 (+ 664us) server_negotiation.cc:407] Connection header received
0923 14:36:41.929655 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.929658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.929696 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.929738 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.930465 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.931096 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.932185 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.932472 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.932536 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.933722 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.933726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.933730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.933753 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.935101 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.935591 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.935594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.935596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.935674 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.935990 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.935993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.935994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.936147 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.936222 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.936547 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.936626 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:36:41.937131 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.928940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:45373 (local address 127.0.0.1:48746)
0923 14:36:41.929463 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.929473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.929568 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.929854 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.929856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.930215 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.930337 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.930343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.931205 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.931207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.932087 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.932093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.932653 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.932656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.932753 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.933247 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.933257 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.933618 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.935202 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.935207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.935208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.935495 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.935776 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.935779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.935781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.935890 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.936266 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.936269 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.936384 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.936843 (+ 459us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.936986 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":96,"spinlock_wait_cycles":8576,"thread_start_us":71,"threads_started":1}
Sep 23 14:36:41 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174200, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.62@KRBTEST.COM for oryx/127.2.218.60@KRBTEST.COM
I20260923 14:36:41.947963 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 9100
I20260923 14:36:41.954160 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:39005
--local_ip_for_outbound_sockets=127.2.218.1
--tserver_master_addrs=127.2.218.62:39787
--webserver_port=45525
--webserver_interface=127.2.218.1
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab
--principal=oryx/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.218.62:39787,127.2.218.61:45373
--tserver_master_addrs=127.2.218.62:39787,127.2.218.61:45373,127.2.218.60:33311 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:36:41.955559 9762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:41.955674 9762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } }
I20260923 14:36:41.957023 9762 leader_election.cc:288] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787), b007bc31c2df4885ada07189c37f05ce (127.2.218.61:45373)
I20260923 14:36:41.957413 9048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "458bac8acab4452e921aa9185b006045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" is_pre_election: true
I20260923 14:36:41.958240 9645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "458bac8acab4452e921aa9185b006045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b007bc31c2df4885ada07189c37f05ce" is_pre_election: true
I20260923 14:36:41.958395 9645 raft_consensus.cc:1799] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 LEARNER]: Handling vote request from an unknown peer 458bac8acab4452e921aa9185b006045
I20260923 14:36:41.958503 9645 raft_consensus.cc:2396] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 458bac8acab4452e921aa9185b006045 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260923 14:36:41.958635 9759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.928983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.2.218.60:33311)
0923 14:36:41.929057 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.929060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.929272 (+ 212us) server_negotiation.cc:407] Connection header received
0923 14:36:41.929390 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.929392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.929420 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.929455 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.930205 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.930905 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.932106 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.932618 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.932685 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.955672 (+ 22987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.955676 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.955678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.955700 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.957010 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.957446 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.957448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.957450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.957518 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.957811 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.957814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.957815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.958016 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.958077 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.958402 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.958505 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":29}
I20260923 14:36:41.958809 9708 leader_election.cc:302] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 458bac8acab4452e921aa9185b006045; no voters: b007bc31c2df4885ada07189c37f05ce, d24f06a6135f49f2bdaf29e7f60bb586
I20260923 14:36:41.959046 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.928734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:33311 (local address 127.0.0.1:40010)
0923 14:36:41.929202 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.929214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.929315 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.929752 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.929755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.929968 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.930071 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.930077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.931200 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.931202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.932005 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.932014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.932792 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.932795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.933073 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.954946 (+ 21873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.954962 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.955560 (+ 598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.957111 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.957116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.957118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.957319 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.957631 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.957634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.957637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.957715 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.958156 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.958161 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.958243 (+ 82us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.958488 (+ 245us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.958888 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":21,"thread_start_us":90,"threads_started":1}
I20260923 14:36:41.959159 9762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:41.960107 9762 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 14:36:41.960613 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.939706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:33311 (local address 127.0.0.1:40012)
0923 14:36:41.939810 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.939821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.939883 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.940143 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.940145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.940296 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.940372 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.940377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.940973 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.940976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.941729 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.941738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.942086 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.942089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.942193 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.942527 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.942539 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.943578 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.958875 (+ 15297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.958878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.958880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.959087 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.959631 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.959634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.959635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.959711 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.960065 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.960068 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.960165 (+ 97us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.960260 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.960415 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 14:36:41.961251 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.939756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.2.218.60:33311)
0923 14:36:41.939949 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.939956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.939972 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:36:41.940011 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.940012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.940043 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.940103 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.940463 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.940873 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.941826 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.941994 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.942054 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.943704 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.943707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.943709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.943728 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.958600 (+ 14872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.959459 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.959462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.959464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.959541 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.959788 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.959790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.959791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.959973 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.960065 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.960900 (+ 835us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.961043 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260923 14:36:41.962198 9038 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.218.60:33311: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "458bac8acab4452e921aa9185b006045"
member_type: NON_VOTER
last_known_addr {
host: "127.2.218.60"
port: 33311
}
attrs {
promote: true
}
}
dest_uuid: "d24f06a6135f49f2bdaf29e7f60bb586"
cas_config_opid_index: 5
I20260923 14:36:41.962693 9038 consensus_queue.cc:237] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } }
I20260923 14:36:41.963785 9597 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 458bac8acab4452e921aa9185b006045 (127.2.218.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:41.964924 9597 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.218.60:33311
I20260923 14:36:41.965070 9597 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:41.965164 9597 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:41.965575 9689 master.cc:537] Master@127.2.218.60:33311 shutting down...
I20260923 14:36:41.965931 9731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Refusing update from remote peer d24f06a6135f49f2bdaf29e7f60bb586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 14:36:41.966518 9596 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 14:36:41.969228 9023 consensus_peers.cc:487] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 -> Peer 458bac8acab4452e921aa9185b006045 (127.2.218.60:33311): Couldn't send request to peer 458bac8acab4452e921aa9185b006045. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 14:36:41.970228 9689 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:41.970433 9689 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:41.970512 9689 tablet_replica.cc:330] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: stopping tablet replica
I20260923 14:36:41.972388 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.963675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:45373 (local address 127.0.0.1:48756)
0923 14:36:41.967424 (+ 3749us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.967439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.967512 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.968558 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.968561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.968580 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:41.968648 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.968653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.969894 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.969898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.971142 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.971152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.972062 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.972065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.972143 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.972177 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.972231 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.972285 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3664}
I20260923 14:36:41.972718 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.963760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.2.218.61:45373)
0923 14:36:41.963860 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.963865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.967532 (+ 3667us) server_negotiation.cc:407] Connection header received
0923 14:36:41.967575 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.967578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.967622 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.967698 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:41.968768 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.969452 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.971263 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.971812 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.971877 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.971983 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.972569 (+ 586us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.972629 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:36:41.973344 9645 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 LEARNER]: Refusing update from remote peer d24f06a6135f49f2bdaf29e7f60bb586: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 14:36:41.973845 9596 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 14:36:41.974741 9689 master.cc:552] Master@127.2.218.60:33311 shutdown complete.
I20260923 14:36:41.974781 9689 master_runner.cc:305] Clearing existing system tablet
I20260923 14:36:41.974963 9689 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:41.974997 9689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:41.975585 9645 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 458bac8acab4452e921aa9185b006045 (127.2.218.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:41.976073 9689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 14:36:41.976750 9786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:41.976929 9689 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:41.977013 9689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:41.977038 9689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal
uuid: "458bac8acab4452e921aa9185b006045"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "a5259565ff705d05651bf047b8df3d82"
server_key_iv: "3cc250befb9ef2857ffc165ab63a2555"
server_key_version: "encryptionkey@0"
I20260923 14:36:41.977074 9675 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader d24f06a6135f49f2bdaf29e7f60bb586. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:41.977162 9675 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: This master's current role is: LEARNER
I20260923 14:36:41.977221 9675 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: NON_VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: true } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:41.977298 9675 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: This master's current role is: LEARNER
I20260923 14:36:41.978600 9596 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586: attempting to promote NON_VOTER b007bc31c2df4885ada07189c37f05ce to VOTER
I20260923 14:36:41.978669 9689 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:41.978941 9596 consensus_queue.cc:237] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } }
I20260923 14:36:41.980324 9645 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 LEARNER]: Refusing update from remote peer d24f06a6135f49f2bdaf29e7f60bb586: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 14:36:41.980489 9689 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 14:36:41.980561 9689 log.cc:1196] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 14:36:41.980686 9597 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 14:36:41.980760 9689 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Deleting consensus metadata
I20260923 14:36:41.980899 9689 master_runner.cc:315] Copying system tablet from 127.2.218.62:39787
I20260923 14:36:41.981647 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.979728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:33311 (local address 127.0.0.1:40026)
0923 14:36:41.981530 (+ 1802us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.981591 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1748}
I20260923 14:36:41.981740 9689 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: tablet copy: Beginning tablet copy session from remote peer at address 127.2.218.62:39787
I20260923 14:36:41.982461 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.982247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:33311 (local address 127.0.0.1:40034)
0923 14:36:41.982365 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.982406 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 14:36:41.982640 9023 consensus_peers.cc:487] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 -> Peer 458bac8acab4452e921aa9185b006045 (127.2.218.60:33311): Couldn't send request to peer 458bac8acab4452e921aa9185b006045. Status: Network error: Client connection negotiation failed: client connection to 127.2.218.60:33311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 14:36:41.983476 9597 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b007bc31c2df4885ada07189c37f05ce (127.2.218.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:41.983875 9645 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b007bc31c2df4885ada07189c37f05ce (127.2.218.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:41.984700 9597 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:41.984787 9597 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:41.984884 9597 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:41.984939 9597 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:41.985390 9787 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:41.985486 9787 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:36:41.993180 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.982493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.2.218.62:39787)
0923 14:36:41.982584 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:41.982588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:41.986311 (+ 3723us) server_negotiation.cc:407] Connection header received
0923 14:36:41.986383 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:41.986386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:41.986421 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:41.986490 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:41.986843 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.987465 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.988454 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.988738 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.988838 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.989920 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:41.989923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:41.989925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:41.989947 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:41.991403 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.991978 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.991984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.991987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.992076 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:41.992407 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:41.992413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:41.992417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:41.992637 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:41.992741 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:41.992983 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 14:36:41.993075 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:36:41.993443 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:41.982658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:34716)
0923 14:36:41.986232 (+ 3574us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:41.986244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:41.986323 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:41.986515 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:41.986518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:41.986664 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:41.986744 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.986749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.987563 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.987565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.988344 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:41.988353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:41.988855 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:41.988858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:41.988943 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:41.989425 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:41.989438 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:41.989792 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:41.991516 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.991519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.991521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.991868 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.992210 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:41.992213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:41.992215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:41.992303 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:41.992750 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:41.992753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:41.992832 (+ 79us) client_negotiation.cc:769] Sending connection context
0923 14:36:41.992927 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 14:36:41.993039 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3519,"thread_start_us":53,"threads_started":1}
I20260923 14:36:41.993765 9058 tablet_copy_service.cc:140] P d24f06a6135f49f2bdaf29e7f60bb586: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 458bac8acab4452e921aa9185b006045 ({username='oryx', principal='oryx/127.2.218.60@KRBTEST.COM'} at 127.0.0.1:34716)
I20260923 14:36:41.993844 9058 tablet_copy_service.cc:161] P d24f06a6135f49f2bdaf29e7f60bb586: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 458bac8acab4452e921aa9185b006045 at {username='oryx', principal='oryx/127.2.218.60@KRBTEST.COM'} at 127.0.0.1:34716: session id = 458bac8acab4452e921aa9185b006045-00000000000000000000000000000000
I20260923 14:36:41.994771 9058 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 14:36:41.995816 9689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 14:36:41.997596 9689 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: tablet copy: Starting download of 0 data blocks...
I20260923 14:36:41.997742 9689 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: tablet copy: Starting download of 1 WAL segments...
I20260923 14:36:41.998936 9689 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 14:36:42.000406 9058 tablet_copy_service.cc:342] P d24f06a6135f49f2bdaf29e7f60bb586: Request end of tablet copy session 458bac8acab4452e921aa9185b006045-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.218.60@KRBTEST.COM'} at 127.0.0.1:34716
I20260923 14:36:42.000473 9058 tablet_copy_service.cc:433] P d24f06a6135f49f2bdaf29e7f60bb586: ending tablet copy session 458bac8acab4452e921aa9185b006045-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 458bac8acab4452e921aa9185b006045
I20260923 14:36:42.006695 9689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:42.008090 9796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:42.010929 9797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:42.011077 9799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:42.014295 9689 server_base.cc:1034] running on GCE node
I20260923 14:36:42.014469 9689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:42.014729 9689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:42.015873 9689 hybrid_clock.cc:648] HybridClock initialized: now 1790174202015854 us; error 33 us; skew 500 ppm
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:42.017203 9689 init.cc:377] Logged in from keytab as oryx/127.2.218.60@KRBTEST.COM (short username oryx)
I20260923 14:36:42.017549 9689 webserver.cc:460] Webserver started at http://127.2.218.60:38475/ using document root <none> and password file <none>
I20260923 14:36:42.017700 9689 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:42.017735 9689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:42.018952 9689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:42.019804 9806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:42.019986 9689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:42.020049 9689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal
uuid: "458bac8acab4452e921aa9185b006045"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "a5259565ff705d05651bf047b8df3d82"
server_key_iv: "3cc250befb9ef2857ffc165ab63a2555"
server_key_version: "encryptionkey@0"
I20260923 14:36:42.020116 9689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:42.031291 9689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:42.032992 9689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:42.037386 9689 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.60:33311
I20260923 14:36:42.037959 9689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
I20260923 14:36:42.039408 9858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.60:33311 every 8 connection(s)
I20260923 14:36:42.041342 9859 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:36:42.042341 9859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Bootstrap starting.
I20260923 14:36:42.052954 9859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:36:42.053211 9859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Bootstrap complete.
I20260923 14:36:42.053730 9859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } }
I20260923 14:36:42.053864 9859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 458bac8acab4452e921aa9185b006045, State: Initialized, Role: LEARNER
I20260923 14:36:42.054006 9859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } }
I20260923 14:36:42.054323 9862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:42.054479 9859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:42.054509 9862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: This master's current role is: LEARNER
I20260923 14:36:42.057085 9873 catalog_manager.cc:1259] Loaded cluster ID: 7a542e1571b642c9817a2e9f29ac9915
I20260923 14:36:42.057186 9873 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: loading cluster ID for follower catalog manager: success
I20260923 14:36:42.059134 9873 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: acquiring CA information for follower catalog manager: success
I20260923 14:36:42.060259 9873 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 14:36:42.072127 9783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:42.072316 9783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:42.072350 9783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:42.075928 9783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:42.075990 9783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:42.076089 9783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:42.080782 9783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab
--principal=oryx/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:39005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=45525
--tserver_master_addrs=127.2.218.62:39787,127.2.218.61:45373,127.2.218.60:33311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:42.081589 9783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:42.082448 9783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:42.090276 9881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:42.090335 9878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:42.090430 9783 server_base.cc:1034] running on GCE node
W20260923 14:36:42.090337 9879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:42.090798 9783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:42.091298 9783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:42.092459 9783 hybrid_clock.cc:648] HybridClock initialized: now 1790174202092437 us; error 37 us; skew 500 ppm
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:42.094640 9783 init.cc:377] Logged in from keytab as oryx/127.2.218.1@KRBTEST.COM (short username oryx)
I20260923 14:36:42.095273 9783 webserver.cc:460] Webserver started at http://127.2.218.1:45525/ using document root <none> and password file <none>
I20260923 14:36:42.095670 9783 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:42.095728 9783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:42.098773 9783 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:42.100448 9888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:42.101459 9783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:42.101558 9783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "50c6accd5b564cc5a4cccba961f80355"
format_stamp: "Formatted at 2026-09-23 14:36:40 on dist-test-slave-zl47"
server_key: "d11088f5f234b4b32d304412b4104287"
server_key_iv: "50e99622dda2a0e8c2c11fc9267e848e"
server_key_version: "encryptionkey@0"
I20260923 14:36:42.102026 9783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:42.122565 9783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:42.124806 9783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:42.124961 9783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:42.125406 9783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:42.126173 9783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:42.126212 9783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:42.126261 9783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:42.126289 9783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:42.133921 9783 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:39005
I20260923 14:36:42.133977 10001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:39005 every 8 connection(s)
I20260923 14:36:42.134761 9783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.60@KRBTEST.COM
I20260923 14:36:42.144147 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 9783
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:42.144351 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 9234
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.61@KRBTEST.COM
I20260923 14:36:42.149283 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.136205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:60389 (local address 127.2.218.61:45373)
0923 14:36:42.136335 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.136340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.137100 (+ 760us) server_negotiation.cc:407] Connection header received
0923 14:36:42.138195 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.138198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.138249 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.138321 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.139622 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.140272 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.141757 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.141950 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.141999 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.145706 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.145711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.145714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.145737 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.147486 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.148087 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.148091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.148092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.148162 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.148423 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.148429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.148431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.148614 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.148670 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.149051 (+ 381us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.149150 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:36:42.149300 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.137550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:57183 (local address 127.2.218.62:39787)
0923 14:36:42.137667 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.137677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.137750 (+ 73us) server_negotiation.cc:407] Connection header received
0923 14:36:42.138440 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.138443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.138486 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.138568 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.139584 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.140281 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.141752 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.142020 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.142138 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.145510 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.145513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.145515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.145541 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.147551 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.148012 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.148018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.148022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.148098 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.148444 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.148446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.148448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.148608 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.148671 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.149053 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.149150 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:36:42.150066 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.136346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:43953 (local address 127.2.218.60:33311)
0923 14:36:42.136577 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.136581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.137099 (+ 518us) server_negotiation.cc:407] Connection header received
0923 14:36:42.138272 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.138277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.138324 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.138396 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.139606 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.140259 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.141423 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.141663 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.141760 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.144417 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.144423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.144429 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.144459 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.146082 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.146997 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.147001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.147003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.147101 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.147468 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.147472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.147473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.147673 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.147760 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.149720 (+ 1960us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.149846 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
I20260923 14:36:42.150427 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.136592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:33311 (local address 127.2.218.1:43953)
0923 14:36:42.136978 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.137021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.138009 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.138478 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.138486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.139010 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:42.139474 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.139479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.140379 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.140382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.141307 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.141315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.141885 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.141888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.142046 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.142609 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:42.142639 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:42.144213 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:42.146536 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.146542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.146554 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.146879 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.147200 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.147202 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.147205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.147363 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.147778 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:42.147785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:42.148029 (+ 244us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.148192 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.148464 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":214,"spinlock_wait_cycles":37760,"thread_start_us":115,"threads_started":1}
I20260923 14:36:42.150640 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.137441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.1:57183)
0923 14:36:42.137661 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.137678 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.138013 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.138583 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.138586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.139011 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:42.139463 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.139474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.140391 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.140394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.141622 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.141632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.142172 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.142175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.142428 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.142993 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:42.143025 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:42.145365 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:42.147655 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.147658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.147659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.147918 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.148240 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.148243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.148245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.148356 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.148742 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:42.148743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:42.148850 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.149506 (+ 656us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.150430 (+ 924us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1210,"spinlock_wait_cycles":11648,"thread_start_us":83,"threads_started":1}
I20260923 14:36:42.151103 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.136591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:45373 (local address 127.2.218.1:60389)
0923 14:36:42.136978 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.137021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.138013 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.138478 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.138486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.139010 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:42.139472 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.139479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.140391 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.140394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.141622 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.141632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.142156 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.142162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.142424 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.143063 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:42.143077 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:42.145569 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:42.147800 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.147804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.147808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.147998 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.148263 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.148265 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.148268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.148328 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.148737 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:42.148740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:42.148850 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.149401 (+ 551us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.150305 (+ 904us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":362,"spinlock_wait_cycles":37120,"thread_start_us":127,"threads_started":1}
I20260923 14:36:42.151537 10002 heartbeater.cc:347] Connected to a master server at 127.2.218.62:39787
I20260923 14:36:42.151556 10003 heartbeater.cc:347] Connected to a master server at 127.2.218.60:33311
I20260923 14:36:42.151722 10002 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:42.151722 10003 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:42.151822 10004 heartbeater.cc:347] Connected to a master server at 127.2.218.61:45373
I20260923 14:36:42.151872 10004 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:42.152225 10003 heartbeater.cc:511] Master 127.2.218.60:33311 requested a full tablet report, sending...
I20260923 14:36:42.152226 10004 heartbeater.cc:511] Master 127.2.218.61:45373 requested a full tablet report, sending...
I20260923 14:36:42.152287 10002 heartbeater.cc:511] Master 127.2.218.62:39787 requested a full tablet report, sending...
I20260923 14:36:42.152661 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:46199
--local_ip_for_outbound_sockets=127.2.218.2
--tserver_master_addrs=127.2.218.62:39787
--webserver_port=34673
--webserver_interface=127.2.218.2
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab
--principal=oryx/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.218.62:39787,127.2.218.61:45373
--tserver_master_addrs=127.2.218.62:39787,127.2.218.61:45373,127.2.218.60:33311 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:36:42.153515 9635 ts_manager.cc:194] Registered new tserver with Master: 50c6accd5b564cc5a4cccba961f80355 (127.2.218.1:39005)
I20260923 14:36:42.153578 9814 ts_manager.cc:194] Registered new tserver with Master: 50c6accd5b564cc5a4cccba961f80355 (127.2.218.1:39005)
I20260923 14:36:42.154313 9038 ts_manager.cc:194] Re-registered known tserver with Master: 50c6accd5b564cc5a4cccba961f80355 (127.2.218.1:39005)
I20260923 14:36:42.154835 9038 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:57183
W20260923 14:36:42.247383 10010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:42.247642 10010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:42.247681 10010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:42.251142 10010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:42.251181 10010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:42.251227 10010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:42.255589 10010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab
--principal=oryx/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:46199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=34673
--tserver_master_addrs=127.2.218.62:39787,127.2.218.61:45373,127.2.218.60:33311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:42.256325 10010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:42.257014 10010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:42.264528 10015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:42.264529 10018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:42.264528 10016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:42.264570 10010 server_base.cc:1034] running on GCE node
I20260923 14:36:42.265087 10010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:42.265573 10010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:42.266764 10010 hybrid_clock.cc:648] HybridClock initialized: now 1790174202266757 us; error 37 us; skew 500 ppm
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:42.268927 10010 init.cc:377] Logged in from keytab as oryx/127.2.218.2@KRBTEST.COM (short username oryx)
I20260923 14:36:42.269557 10010 webserver.cc:460] Webserver started at http://127.2.218.2:34673/ using document root <none> and password file <none>
I20260923 14:36:42.269989 10010 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:42.270079 10010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:42.272943 10010 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:36:42.274549 10025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:42.275493 10010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:42.275609 10010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "b476e81a19fc420bb11e0666879be1e7"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "67e9894275282f5268de44097f8d7431"
server_key_iv: "575457b68116c0fd9a6bdb8f45dc2134"
server_key_version: "encryptionkey@0"
I20260923 14:36:42.276002 10010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:42.290328 10010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:42.292542 10010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:42.292707 10010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:42.293141 10010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:42.293789 10010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:42.293856 10010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:42.293912 10010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:42.293951 10010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:42.301267 10010 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:46199
I20260923 14:36:42.301292 10138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:46199 every 8 connection(s)
I20260923 14:36:42.302001 10010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:36:42.309201 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 10010
I20260923 14:36:42.309365 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 9368
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.61@KRBTEST.COM
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.60@KRBTEST.COM
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:42.316184 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.303418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:55367 (local address 127.2.218.61:45373)
0923 14:36:42.303544 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.303549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.304242 (+ 693us) server_negotiation.cc:407] Connection header received
0923 14:36:42.305196 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.305201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.305247 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.305318 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.306550 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.307005 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.308380 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.308566 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.308617 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.312138 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.312142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.312144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.312170 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.313697 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.314358 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.314362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.314363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.314462 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.314914 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.314917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.314919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.315323 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.315416 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.315953 (+ 537us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.316050 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:36:42.316184 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.303902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:49697 (local address 127.2.218.60:33311)
0923 14:36:42.304009 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.304013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.304851 (+ 838us) server_negotiation.cc:407] Connection header received
0923 14:36:42.305212 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.305214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.305255 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.305305 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.306528 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.307225 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.308372 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.308641 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.308733 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.312288 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.312292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.312295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.312321 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.313836 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.314358 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.314362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.314363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.314558 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.314914 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.314917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.314919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.315117 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.315176 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.315940 (+ 764us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.316050 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:36:42.316738 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.304675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:42441 (local address 127.2.218.62:39787)
0923 14:36:42.304794 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.304798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.305012 (+ 214us) server_negotiation.cc:407] Connection header received
0923 14:36:42.305188 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.305191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.305240 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.305308 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.306558 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.307250 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.308081 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.308289 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.308387 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.313255 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.313261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.313266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.313296 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.315040 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.315515 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.315520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.315524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.315602 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.315882 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.315887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.315892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.316101 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.316181 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.316498 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.316609 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 14:36:42.316887 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:37507
--local_ip_for_outbound_sockets=127.2.218.3
--tserver_master_addrs=127.2.218.62:39787
--webserver_port=44759
--webserver_interface=127.2.218.3
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab
--principal=oryx/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.218.62:39787,127.2.218.61:45373
--tserver_master_addrs=127.2.218.62:39787,127.2.218.61:45373,127.2.218.60:33311 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:36:42.317397 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.303722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:45373 (local address 127.2.218.2:55367)
0923 14:36:42.304107 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.304147 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.304987 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.305479 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.305481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.305911 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:42.306410 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.306423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.307131 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.307132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.308259 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.308269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.308760 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.308762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.308859 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.309841 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:42.309862 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:42.311766 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:42.313836 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.313841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.313853 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.314254 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.314699 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.314704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.314708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.314815 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.315427 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:42.315434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:42.315702 (+ 268us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.316599 (+ 897us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.316687 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":226,"spinlock_wait_cycles":12288,"thread_start_us":112,"threads_started":1}
I20260923 14:36:42.317588 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.304541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.2:42441)
0923 14:36:42.304925 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.304936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.305023 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.305465 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.305472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.305915 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:42.306408 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.306422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.307366 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.307369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.307970 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.307977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.308513 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.308516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.308682 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.310916 (+ 2234us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:42.310934 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:42.312382 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:42.315178 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.315194 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.315199 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.315415 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.315700 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.315705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.315709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.315790 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.316249 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:42.316254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:42.316338 (+ 84us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.316855 (+ 517us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.316976 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":94,"spinlock_wait_cycles":5760,"thread_start_us":80,"threads_started":1}
I20260923 14:36:42.317394 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.303721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:33311 (local address 127.2.218.2:49697)
0923 14:36:42.304763 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.304774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.304999 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.305467 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.305476 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.305911 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:42.306407 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.306419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.307333 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.307335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.308264 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.308269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.308752 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.308756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.308896 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.310163 (+ 1267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:42.310175 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:42.312132 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:42.313934 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.313938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.313939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.314254 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.314644 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.314649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.314654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.314815 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.315459 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:42.315461 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:42.315702 (+ 241us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.316326 (+ 624us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.316599 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":289,"spinlock_wait_cycles":14720,"thread_start_us":86,"threads_started":1}
I20260923 14:36:42.318765 10140 heartbeater.cc:347] Connected to a master server at 127.2.218.60:33311
I20260923 14:36:42.318789 10141 heartbeater.cc:347] Connected to a master server at 127.2.218.61:45373
I20260923 14:36:42.318787 10139 heartbeater.cc:347] Connected to a master server at 127.2.218.62:39787
I20260923 14:36:42.318953 10141 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:42.318951 10140 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:42.318950 10139 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:42.319460 10139 heartbeater.cc:511] Master 127.2.218.62:39787 requested a full tablet report, sending...
I20260923 14:36:42.319523 10140 heartbeater.cc:511] Master 127.2.218.60:33311 requested a full tablet report, sending...
I20260923 14:36:42.319458 10141 heartbeater.cc:511] Master 127.2.218.61:45373 requested a full tablet report, sending...
I20260923 14:36:42.320449 9038 ts_manager.cc:194] Re-registered known tserver with Master: b476e81a19fc420bb11e0666879be1e7 (127.2.218.2:46199)
I20260923 14:36:42.320624 9635 ts_manager.cc:194] Registered new tserver with Master: b476e81a19fc420bb11e0666879be1e7 (127.2.218.2:46199)
I20260923 14:36:42.321096 9038 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:42441
I20260923 14:36:42.321592 9814 ts_manager.cc:194] Registered new tserver with Master: b476e81a19fc420bb11e0666879be1e7 (127.2.218.2:46199)
I20260923 14:36:42.361615 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.358545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:33311 (local address 127.0.0.1:40046)
0923 14:36:42.358653 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.358668 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.358742 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.358971 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.358973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.358984 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:42.359048 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.359052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.359891 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.359892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.360584 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.360592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.361375 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.361377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.361431 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.361447 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.361487 (+ 40us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.361531 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 14:36:42.361667 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.358608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.2.218.60:33311)
0923 14:36:42.358708 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.358712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.358734 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:36:42.358823 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.358825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.358874 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.358959 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:42.359134 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.359794 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.360698 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.361270 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.361373 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.361489 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.361519 (+ 30us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.361571 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:36:42.362296 9833 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 14:36:42.362612 9862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d24f06a6135f49f2bdaf29e7f60bb586. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:42.362771 9862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: This master's current role is: LEARNER
W20260923 14:36:42.411659 10146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:42.411850 10146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:42.411885 10146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:42.415290 10146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:42.415328 10146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:42.415408 10146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:42.420725 10146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab
--principal=oryx/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:37507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=44759
--tserver_master_addrs=127.2.218.62:39787,127.2.218.61:45373,127.2.218.60:33311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:42.421556 10146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:42.422398 10146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:42.430197 10154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:42.430199 10152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:42.430197 10151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:42.430364 10146 server_base.cc:1034] running on GCE node
I20260923 14:36:42.430724 10146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:42.431191 10146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:42.432337 10146 hybrid_clock.cc:648] HybridClock initialized: now 1790174202432320 us; error 32 us; skew 500 ppm
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:42.434657 10146 init.cc:377] Logged in from keytab as oryx/127.2.218.3@KRBTEST.COM (short username oryx)
I20260923 14:36:42.435529 10146 webserver.cc:460] Webserver started at http://127.2.218.3:44759/ using document root <none> and password file <none>
I20260923 14:36:42.436010 10146 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:42.436080 10146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:42.439157 10146 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:36:42.440722 10161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:42.441728 10146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:42.441849 10146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "c68f17b782494aed8dcec316efb7a76f"
format_stamp: "Formatted at 2026-09-23 14:36:41 on dist-test-slave-zl47"
server_key: "fb90aa5d5b9d38675651632740c75b18"
server_key_iv: "2c27ee5d0bc933068837ff05e17112c0"
server_key_version: "encryptionkey@0"
I20260923 14:36:42.442304 10146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:42.454052 10146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:42.456774 10146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:42.456936 10146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:42.457386 10146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:42.458057 10146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:42.458153 10146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:42.458226 10146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:42.458269 10146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:42.466559 10146 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:37507
I20260923 14:36:42.466607 10274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:37507 every 8 connection(s)
I20260923 14:36:42.467547 10146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:36:42.473598 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 10146
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.61@KRBTEST.COM
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
Sep 23 14:36:42 dist-test-slave-zl47 krb5kdc[8993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174202, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.60@KRBTEST.COM
I20260923 14:36:42.483778 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.470342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.3:45627)
0923 14:36:42.470567 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.470577 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.471140 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.471605 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.471613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.472047 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:42.472454 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.472462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.473411 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.473414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.474516 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.474526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.475100 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.475106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.475370 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.476672 (+ 1302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:42.476684 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:42.478430 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:42.480805 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.480810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.480821 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.481135 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.481493 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.481496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.481498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.481633 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.482028 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:42.482034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:42.482362 (+ 328us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.482568 (+ 206us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.482866 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":316,"spinlock_wait_cycles":12288,"thread_start_us":108,"threads_started":1}
I20260923 14:36:42.484197 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.469083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:58777 (local address 127.2.218.61:45373)
0923 14:36:42.469249 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.469254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.470303 (+ 1049us) server_negotiation.cc:407] Connection header received
0923 14:36:42.471344 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.471347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.471402 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.471475 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.472572 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.473073 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.474667 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.474915 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.474990 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.478782 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.478787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.478795 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.478823 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.481983 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.482537 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.482540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.482542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.482640 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.482954 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.482957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.482959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.483157 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.483252 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.483902 (+ 650us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.484025 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:36:42.484429 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.471110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:45627 (local address 127.2.218.62:39787)
0923 14:36:42.471246 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.471251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.471263 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:42.471367 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.471369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.471418 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.471485 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.472577 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.473298 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.474612 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.474890 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.474966 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.478556 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.478559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.478563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.478589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.480682 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.481261 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.481267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.481271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.481379 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.481743 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.481750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.481752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.481924 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.482032 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.484174 (+ 2142us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.484291 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:36:42.484463 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.469716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:45373 (local address 127.2.218.3:58777)
0923 14:36:42.470167 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.470214 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.471138 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.471614 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.471621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.472023 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:42.472441 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.472453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.473213 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.473216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.474516 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.474529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.475139 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.475142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.475367 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.475974 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:42.475999 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:42.478672 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:42.482147 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.482152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.482154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.482415 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.482758 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.482763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.482766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.482851 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.483305 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:42.483307 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:42.483421 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.483519 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.483663 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":115,"thread_start_us":108,"threads_started":1}
I20260923 14:36:42.485387 10275 heartbeater.cc:347] Connected to a master server at 127.2.218.62:39787
I20260923 14:36:42.485591 10275 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:42.486080 10275 heartbeater.cc:511] Master 127.2.218.62:39787 requested a full tablet report, sending...
I20260923 14:36:42.486631 10277 heartbeater.cc:347] Connected to a master server at 127.2.218.61:45373
I20260923 14:36:42.486725 10277 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:42.486913 10277 heartbeater.cc:511] Master 127.2.218.61:45373 requested a full tablet report, sending...
I20260923 14:36:42.487545 9038 ts_manager.cc:194] Re-registered known tserver with Master: c68f17b782494aed8dcec316efb7a76f (127.2.218.3:37507)
I20260923 14:36:42.488157 9635 ts_manager.cc:194] Registered new tserver with Master: c68f17b782494aed8dcec316efb7a76f (127.2.218.3:37507)
I20260923 14:36:42.488247 9038 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:45627
I20260923 14:36:42.489293 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.475956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.2.218.60:33311)
0923 14:36:42.476396 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.476402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.478007 (+ 1605us) server_negotiation.cc:407] Connection header received
0923 14:36:42.478128 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.478131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.478175 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.478291 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.480354 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.481019 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.482364 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.482729 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.482807 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.484724 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.484728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.484730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.484752 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.487568 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.488018 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.488020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.488022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.488090 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.488391 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.488393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.488395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.488584 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.488675 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.489013 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.489124 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":74,"threads_started":1}
I20260923 14:36:42.490748 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.471126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:33497 (local address 127.2.218.60:33311)
0923 14:36:42.471234 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.471238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.471253 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:36:42.471497 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.471500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.471541 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.471586 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.472576 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.473289 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.474712 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.474988 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.475049 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.480331 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.480334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.480336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.480356 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.484018 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.486562 (+ 2544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.486565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.486567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.486671 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.489906 (+ 3235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.489909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.489910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.490119 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.490198 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.490498 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.490590 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 14:36:42.491217 10281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.470762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:33311 (local address 127.2.218.3:33497)
0923 14:36:42.470988 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.470998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.471398 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.471722 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.471725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.472054 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:42.472456 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.472462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.473411 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.473414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.474628 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.474632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.475185 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.475188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.475371 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.476325 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:42.476337 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:42.480207 (+ 3870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:42.486177 (+ 5970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.486182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.486185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.486431 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.489729 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:42.489731 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:42.489732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:42.489805 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:42.490248 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:42.490250 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:42.490329 (+ 79us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.490908 (+ 579us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.491066 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":508,"spinlock_wait_cycles":26368,"thread_start_us":94,"threads_started":1}
I20260923 14:36:42.491402 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.475519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.2.218.62:39787)
0923 14:36:42.475758 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.475771 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.477398 (+ 1627us) server_negotiation.cc:407] Connection header received
0923 14:36:42.477511 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.477515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.477580 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.477667 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.478821 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.479603 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.480699 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.480915 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.481008 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.486012 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.486016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.486018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.486041 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.489620 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.490023 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.490026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.490028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.490126 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.490410 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.490413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.490417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.490611 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.490708 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.491079 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.491210 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260923 14:36:42.491768 10276 heartbeater.cc:347] Connected to a master server at 127.2.218.60:33311
I20260923 14:36:42.491859 10276 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:42.492004 10276 heartbeater.cc:511] Master 127.2.218.60:33311 requested a full tablet report, sending...
I20260923 14:36:42.492547 9814 ts_manager.cc:194] Registered new tserver with Master: c68f17b782494aed8dcec316efb7a76f (127.2.218.3:37507)
I20260923 14:36:42.494304 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.477009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.2.218.61:45373)
0923 14:36:42.477302 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.477325 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.478130 (+ 805us) server_negotiation.cc:407] Connection header received
0923 14:36:42.478167 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.478169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.478210 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.478290 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.486558 (+ 8268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.487340 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.488486 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.488781 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.488875 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.490757 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.490761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.490764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.490786 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.492457 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.493005 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.493009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.493013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.493096 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.493387 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.493390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.493391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.493602 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.493716 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.493931 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.494072 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":81,"threads_started":1}
I20260923 14:36:42.495025 9038 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:42.497668 9038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:42.499989 9795 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: NON_VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: true } } }
I20260923 14:36:42.500128 9795 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:42.503077 9795 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586: attempting to promote NON_VOTER 458bac8acab4452e921aa9185b006045 to VOTER
I20260923 14:36:42.503418 9795 consensus_queue.cc:237] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } }
I20260923 14:36:42.504359 9833 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 LEARNER]: Refusing update from remote peer d24f06a6135f49f2bdaf29e7f60bb586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 14:36:42.504535 9645 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Refusing update from remote peer d24f06a6135f49f2bdaf29e7f60bb586: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 14:36:42.504848 10301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 14:36:42.505246 10301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 14:36:42.506436 10293 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 458bac8acab4452e921aa9185b006045 (127.2.218.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:42.507544 10293 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:42.507663 10293 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:42.508129 10294 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:42.508250 10294 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:42.508430 9645 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 458bac8acab4452e921aa9185b006045 (127.2.218.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:42.508477 9833 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 458bac8acab4452e921aa9185b006045 (127.2.218.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:42.508873 10301 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:42.509012 10301 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:42.510036 9862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:42.510186 9862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:36:42.511168 10303 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:42.511284 10303 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:36:42.515278 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.509799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:39005 (local address 127.0.0.1:44308)
0923 14:36:42.510418 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.510434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.510526 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.512172 (+ 1646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.512174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.512190 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:42.512265 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.512271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.513078 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.513083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.514036 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.514056 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.514962 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.514965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.515035 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.515054 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.515109 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.515178 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362}
I20260923 14:36:42.515319 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.510690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.2.218.1:39005)
0923 14:36:42.511845 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.511852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.511868 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:36:42.511919 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.511922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.512036 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.512188 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:42.512345 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.512971 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.514208 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.514844 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.514970 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.515090 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.515134 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.515201 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1079,"thread_start_us":78,"threads_started":1}
I20260923 14:36:42.516016 10307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.511243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:37507 (local address 127.0.0.1:46132)
0923 14:36:42.511473 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.511486 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.511570 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.512206 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.512210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.512230 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:42.512332 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.512339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.513611 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.513614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.514721 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.514730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.515682 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.515686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.515765 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.515788 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.515835 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.515895 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":45,"thread_start_us":76,"threads_started":1}
I20260923 14:36:42.516568 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.511330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.2.218.3:37507)
0923 14:36:42.511776 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.511788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.511810 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:36:42.511866 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.511871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.512037 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.512464 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:42.512569 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.513488 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.514843 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.515555 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.516195 (+ 640us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.516309 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.516355 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.516427 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":62,"threads_started":1}
I20260923 14:36:42.516886 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.509990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:46199 (local address 127.0.0.1:55494)
0923 14:36:42.510361 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.510373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.510436 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.512847 (+ 2411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.512850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.512884 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:42.512961 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.512966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.514317 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.514320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.515196 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.515207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.516527 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.516530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.516633 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.516662 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.516716 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.516777 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260923 14:36:42.517518 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.510649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.2.218.2:46199)
0923 14:36:42.510953 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.510967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.510989 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:36:42.512509 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.512519 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.512696 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.513091 (+ 395us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:42.513219 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.514194 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.515570 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.516385 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.517136 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.517263 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.517309 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.517384 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260923 14:36:42.517839 9936 tablet_service.cc:1467] Processing CreateTablet for tablet 56524581fe6d40339384e3a1e4aa46d4 (DEFAULT_TABLE table=test-table [id=6eeb019825704c79970336658921723a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:42.518723 9936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56524581fe6d40339384e3a1e4aa46d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:42.518747 10209 tablet_service.cc:1467] Processing CreateTablet for tablet 56524581fe6d40339384e3a1e4aa46d4 (DEFAULT_TABLE table=test-table [id=6eeb019825704c79970336658921723a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:42.519533 10209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56524581fe6d40339384e3a1e4aa46d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:42.519721 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 56524581fe6d40339384e3a1e4aa46d4 (DEFAULT_TABLE table=test-table [id=6eeb019825704c79970336658921723a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:42.520538 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56524581fe6d40339384e3a1e4aa46d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:42.525290 10309 tablet_bootstrap.cc:492] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: Bootstrap starting.
I20260923 14:36:42.526320 10310 tablet_bootstrap.cc:492] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: Bootstrap starting.
I20260923 14:36:42.526803 10311 tablet_bootstrap.cc:492] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: Bootstrap starting.
I20260923 14:36:42.527633 10309 tablet_bootstrap.cc:654] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:42.528335 10310 tablet_bootstrap.cc:654] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:42.528483 10309 log.cc:824] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:42.528699 10311 tablet_bootstrap.cc:654] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:42.529136 10310 log.cc:824] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:42.529479 10311 log.cc:824] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:42.530478 10309 tablet_bootstrap.cc:492] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: No bootstrap required, opened a new log
I20260923 14:36:42.530658 10309 ts_tablet_manager.cc:1397] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:36:42.530957 10310 tablet_bootstrap.cc:492] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: No bootstrap required, opened a new log
I20260923 14:36:42.531128 10310 ts_tablet_manager.cc:1397] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:36:42.534891 10309 raft_consensus.cc:348] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } }
I20260923 14:36:42.535123 10309 raft_consensus.cc:374] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:42.535209 10309 raft_consensus.cc:729] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68f17b782494aed8dcec316efb7a76f, State: Initialized, Role: FOLLOWER
I20260923 14:36:42.535239 10310 raft_consensus.cc:348] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } }
I20260923 14:36:42.535450 10310 raft_consensus.cc:374] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:42.535508 10310 raft_consensus.cc:729] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c6accd5b564cc5a4cccba961f80355, State: Initialized, Role: FOLLOWER
I20260923 14:36:42.535619 10309 consensus_queue.cc:260] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } }
I20260923 14:36:42.535864 10310 consensus_queue.cc:260] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } }
I20260923 14:36:42.536144 10275 heartbeater.cc:503] Master 127.2.218.62:39787 was elected leader, sending a full tablet report...
I20260923 14:36:42.536473 10309 ts_tablet_manager.cc:1428] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:36:42.536617 10310 ts_tablet_manager.cc:1428] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 14:36:42.538002 10002 heartbeater.cc:503] Master 127.2.218.62:39787 was elected leader, sending a full tablet report...
I20260923 14:36:42.540766 10311 tablet_bootstrap.cc:492] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: No bootstrap required, opened a new log
I20260923 14:36:42.540946 10311 ts_tablet_manager.cc:1397] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260923 14:36:42.544122 10311 raft_consensus.cc:348] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } }
I20260923 14:36:42.544282 10311 raft_consensus.cc:374] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:42.544337 10311 raft_consensus.cc:729] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b476e81a19fc420bb11e0666879be1e7, State: Initialized, Role: FOLLOWER
I20260923 14:36:42.544700 10311 consensus_queue.cc:260] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } }
I20260923 14:36:42.545277 10139 heartbeater.cc:503] Master 127.2.218.62:39787 was elected leader, sending a full tablet report...
I20260923 14:36:42.545557 10311 ts_tablet_manager.cc:1428] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260923 14:36:42.553391 10142 tablet.cc:2369] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:36:42.636854 10005 tablet.cc:2369] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:36:42.719111 10278 tablet.cc:2369] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:42.738763 10316 raft_consensus.cc:482] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:42.738950 10316 raft_consensus.cc:504] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } }
I20260923 14:36:42.740164 10316 leader_election.cc:288] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c68f17b782494aed8dcec316efb7a76f (127.2.218.3:37507), b476e81a19fc420bb11e0666879be1e7 (127.2.218.2:46199)
I20260923 14:36:42.743654 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.740341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:34243 (local address 127.2.218.3:37507)
0923 14:36:42.740467 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.740472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.740684 (+ 212us) server_negotiation.cc:407] Connection header received
0923 14:36:42.740834 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.740841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.740892 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.740951 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:42.741250 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.741690 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.742797 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.743141 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.743243 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.743310 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.743468 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.743534 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:36:42.743701 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.740363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:46199 (local address 127.2.218.1:37835)
0923 14:36:42.740599 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.740612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.740725 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.741018 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.741020 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.741036 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:42.741142 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.741146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.741822 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.741828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.742648 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.742656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.743271 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.743274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.743379 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.743399 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.743479 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.743534 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":17,"spinlock_wait_cycles":896,"thread_start_us":61,"threads_started":1}
I20260923 14:36:42.743696 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.740531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:37835 (local address 127.2.218.2:46199)
0923 14:36:42.740679 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.740683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.740699 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:36:42.740835 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.740843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.740894 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.740992 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:42.741263 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.741697 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.742764 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.743140 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.743262 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.743329 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.743506 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.743578 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:36:42.743907 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.740277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:37507 (local address 127.2.218.1:34243)
0923 14:36:42.740574 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:42.740592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:42.740713 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:42.740993 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:42.740996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:42.741011 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:42.741132 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.741138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.741798 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.741802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.742660 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:42.742669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.743253 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.743257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:42.743348 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.743378 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 14:36:42.743437 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 14:36:42.743519 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":50,"thread_start_us":74,"threads_started":1}
I20260923 14:36:42.744289 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56524581fe6d40339384e3a1e4aa46d4" candidate_uuid: "50c6accd5b564cc5a4cccba961f80355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b476e81a19fc420bb11e0666879be1e7" is_pre_election: true
I20260923 14:36:42.744571 10093 raft_consensus.cc:2454] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50c6accd5b564cc5a4cccba961f80355 in term 0.
I20260923 14:36:42.744552 10229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56524581fe6d40339384e3a1e4aa46d4" candidate_uuid: "50c6accd5b564cc5a4cccba961f80355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68f17b782494aed8dcec316efb7a76f" is_pre_election: true
I20260923 14:36:42.744838 10229 raft_consensus.cc:2454] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50c6accd5b564cc5a4cccba961f80355 in term 0.
I20260923 14:36:42.744995 9889 leader_election.cc:302] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50c6accd5b564cc5a4cccba961f80355, b476e81a19fc420bb11e0666879be1e7; no voters:
I20260923 14:36:42.745240 10316 raft_consensus.cc:2781] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:42.745314 10316 raft_consensus.cc:482] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:42.745342 10316 raft_consensus.cc:3037] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:42.746373 10316 raft_consensus.cc:504] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } }
I20260923 14:36:42.746692 10316 leader_election.cc:288] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [CANDIDATE]: Term 1 election: Requested vote from peers c68f17b782494aed8dcec316efb7a76f (127.2.218.3:37507), b476e81a19fc420bb11e0666879be1e7 (127.2.218.2:46199)
I20260923 14:36:42.746968 10229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56524581fe6d40339384e3a1e4aa46d4" candidate_uuid: "50c6accd5b564cc5a4cccba961f80355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68f17b782494aed8dcec316efb7a76f"
I20260923 14:36:42.747035 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56524581fe6d40339384e3a1e4aa46d4" candidate_uuid: "50c6accd5b564cc5a4cccba961f80355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b476e81a19fc420bb11e0666879be1e7"
I20260923 14:36:42.747081 10229 raft_consensus.cc:3037] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:42.747162 10093 raft_consensus.cc:3037] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:42.748369 10229 raft_consensus.cc:2454] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50c6accd5b564cc5a4cccba961f80355 in term 1.
I20260923 14:36:42.748369 10093 raft_consensus.cc:2454] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50c6accd5b564cc5a4cccba961f80355 in term 1.
I20260923 14:36:42.748668 9890 leader_election.cc:302] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50c6accd5b564cc5a4cccba961f80355, c68f17b782494aed8dcec316efb7a76f; no voters:
I20260923 14:36:42.748862 10316 raft_consensus.cc:2781] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:42.749054 10316 raft_consensus.cc:686] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 1 LEADER]: Becoming Leader. State: Replica: 50c6accd5b564cc5a4cccba961f80355, State: Running, Role: LEADER
I20260923 14:36:42.749339 10316 consensus_queue.cc:237] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } }
I20260923 14:36:42.752139 9038 catalog_manager.cc:5495] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50c6accd5b564cc5a4cccba961f80355 (127.2.218.1). New cstate: current_term: 1 leader_uuid: "50c6accd5b564cc5a4cccba961f80355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } health_report { overall_health: UNKNOWN } } }
I20260923 14:36:42.802611 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.799820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.2.218.1:39005)
0923 14:36:42.799961 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.799966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.799980 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:36:42.800091 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.800093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.800152 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.800221 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:42.800402 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.800788 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.801520 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.801693 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.801760 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.801977 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:42.802318 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:42.802392 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.802444 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.802480 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:36:42.807358 10093 raft_consensus.cc:1261] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7 [term 1 FOLLOWER]: Refusing update from remote peer 50c6accd5b564cc5a4cccba961f80355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:42.807358 10229 raft_consensus.cc:1261] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f [term 1 FOLLOWER]: Refusing update from remote peer 50c6accd5b564cc5a4cccba961f80355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:42.807996 10320 consensus_queue.cc:1035] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:42.808152 10316 consensus_queue.cc:1035] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:42.818702 10324 mvcc.cc:205] Tried to move back new op lower bound from 7332553534693277696 to 7332553534463643648. Current Snapshot: MvccSnapshot[applied={T|T < 7332553534693277696}]
I20260923 14:36:42.822417 9037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:42.822566 9037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:42.826857 9037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d24f06a6135f49f2bdaf29e7f60bb586: Sending DeleteTablet for 3 replicas of tablet 56524581fe6d40339384e3a1e4aa46d4
I20260923 14:36:42.827673 9936 tablet_service.cc:1514] Processing DeleteTablet for tablet 56524581fe6d40339384e3a1e4aa46d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:42 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:44308
I20260923 14:36:42.827693 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 56524581fe6d40339384e3a1e4aa46d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:42 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:55494
I20260923 14:36:42.827702 10209 tablet_service.cc:1514] Processing DeleteTablet for tablet 56524581fe6d40339384e3a1e4aa46d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:42 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:46132
I20260923 14:36:42.828141 10334 tablet_replica.cc:330] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: stopping tablet replica
I20260923 14:36:42.828382 10334 raft_consensus.cc:2229] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:42.828625 10334 raft_consensus.cc:2258] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:42.828889 10332 tablet_replica.cc:330] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: stopping tablet replica
I20260923 14:36:42.829111 10332 raft_consensus.cc:2229] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:42.829331 10332 raft_consensus.cc:2258] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:42.829630 10333 tablet_replica.cc:330] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: stopping tablet replica
I20260923 14:36:42.829842 10333 raft_consensus.cc:2229] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:42.830088 10333 raft_consensus.cc:2258] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:42.830129 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 9006
I20260923 14:36:42.831393 10332 ts_tablet_manager.cc:1905] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:42.833668 10332 ts_tablet_manager.cc:1918] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:42.833757 10332 log.cc:1196] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/56524581fe6d40339384e3a1e4aa46d4
I20260923 14:36:42.834066 10332 ts_tablet_manager.cc:1939] T 56524581fe6d40339384e3a1e4aa46d4 P b476e81a19fc420bb11e0666879be1e7: Deleting consensus metadata
I20260923 14:36:42.835973 10333 ts_tablet_manager.cc:1905] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:42.838187 10333 ts_tablet_manager.cc:1918] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:42.838261 10333 log.cc:1196] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/56524581fe6d40339384e3a1e4aa46d4
I20260923 14:36:42.838541 10333 ts_tablet_manager.cc:1939] T 56524581fe6d40339384e3a1e4aa46d4 P 50c6accd5b564cc5a4cccba961f80355: Deleting consensus metadata
W20260923 14:36:42.839063 10027 connection.cc:535] server connection from 127.0.0.1:55494 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 14:36:42.839290 9889 connection.cc:441] server connection from 127.0.0.1:44308 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44308 (request call id 1) could send its response
I20260923 14:36:42.840683 10334 ts_tablet_manager.cc:1905] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:42.843318 10334 ts_tablet_manager.cc:1918] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:42.843428 10334 log.cc:1196] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/56524581fe6d40339384e3a1e4aa46d4
I20260923 14:36:42.843791 10334 ts_tablet_manager.cc:1939] T 56524581fe6d40339384e3a1e4aa46d4 P c68f17b782494aed8dcec316efb7a76f: Deleting consensus metadata
W20260923 14:36:42.844691 10163 connection.cc:441] server connection from 127.0.0.1:46132 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46132 (request call id 1) could send its response
I20260923 14:36:42.847230 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.829254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.2.218.60:33311)
0923 14:36:42.829344 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.829348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.830855 (+ 1507us) server_negotiation.cc:407] Connection header received
0923 14:36:42.831020 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.831025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.831075 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.831127 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.832239 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.832984 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.834424 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.834819 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.834906 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.837121 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.837128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.837133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.837164 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.838901 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.845367 (+ 6466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.845371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.845372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.845454 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.845757 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.845760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.845762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.845966 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.846075 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.846932 (+ 857us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.847047 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 14:36:42.851150 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.841535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.2.218.60:33311)
0923 14:36:42.841637 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.841641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.842127 (+ 486us) server_negotiation.cc:407] Connection header received
0923 14:36:42.842306 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.842311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.842359 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.842408 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.843842 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.844546 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.845690 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.845968 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.846037 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.847781 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.847785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.847787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.847806 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.849348 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.849901 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.849905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.849906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.849998 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.850308 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.850313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.850316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.850482 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.850629 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.850883 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.850995 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 14:36:42.851531 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:42.841755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.2.218.61:45373)
0923 14:36:42.841862 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:42.841866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:42.842582 (+ 716us) server_negotiation.cc:407] Connection header received
0923 14:36:42.842630 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:42.842632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:42.842693 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:42.842784 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:42.844048 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.844773 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.845864 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:42.846191 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:42.846257 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:42.847880 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:42.847884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:42.847886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:42.847907 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:42.849307 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.849833 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.849839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.849841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.849958 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:42.850341 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:42.850344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:42.850346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:42.850527 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:42.850668 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:42.850973 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 14:36:42.851381 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:36:43.818885 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:43.818503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.2:45281)
0923 14:36:43.818729 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:43.818801 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
W20260923 14:36:43.819249 10139 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:39787 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:43.820775 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:43.820283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.1:40007)
0923 14:36:43.820504 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:43.820675 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
W20260923 14:36:43.821089 10002 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:39787 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:43.821704 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:43.821279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.3:40639)
0923 14:36:43.821518 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:43.821594 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":46,"threads_started":1}
W20260923 14:36:43.822086 10275 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:39787 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:44.327144 10346 raft_consensus.cc:482] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d24f06a6135f49f2bdaf29e7f60bb586)
I20260923 14:36:44.327263 10346 raft_consensus.cc:504] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } }
I20260923 14:36:44.327303 10347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d24f06a6135f49f2bdaf29e7f60bb586)
I20260923 14:36:44.327409 10347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } }
I20260923 14:36:44.327764 10346 leader_election.cc:288] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787), 458bac8acab4452e921aa9185b006045 (127.2.218.60:33311)
I20260923 14:36:44.328130 10347 leader_election.cc:288] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787), b007bc31c2df4885ada07189c37f05ce (127.2.218.61:45373)
I20260923 14:36:44.328320 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.327835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:35208)
0923 14:36:44.328121 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.328225 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":71,"threads_started":1}
W20260923 14:36:44.328504 9619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:44.328671 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.328203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:35224)
0923 14:36:44.328507 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.328579 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":51,"threads_started":1}
W20260923 14:36:44.328827 9807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:44.329310 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.329064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:35234)
0923 14:36:44.329213 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.329254 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":31}
W20260923 14:36:44.329461 9619 leader_election.cc:334] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787): Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:44.329542 10352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.329324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:35246)
0923 14:36:44.329442 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.329488 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 14:36:44.329658 9807 leader_election.cc:334] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787): Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:44.331789 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.327940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:33311 (local address 127.0.0.1:40086)
0923 14:36:44.328421 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.328434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:44.328522 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:44.329292 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:44.329295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:44.329311 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:44.329397 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:44.329404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:44.330203 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:44.330206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:44.330929 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:44.330937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:44.331479 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:44.331486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:44.331538 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:44.331551 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 14:36:44.331613 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 14:36:44.331673 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":50,"threads_started":1}
I20260923 14:36:44.331836 10351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.328099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.2.218.60:33311)
0923 14:36:44.328296 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:44.328300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:44.328596 (+ 296us) server_negotiation.cc:407] Connection header received
0923 14:36:44.328634 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:44.328637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:44.328672 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:44.329197 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:44.329502 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:44.330079 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:44.331025 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:44.331371 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:44.331453 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:44.331569 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:44.331622 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 14:36:44.331696 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":140,"thread_start_us":48,"threads_started":1}
I20260923 14:36:44.331836 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.327974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:45373 (local address 127.0.0.1:48788)
0923 14:36:44.328429 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.328441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:44.328522 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:44.328983 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:44.328986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:44.329004 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:44.329103 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:44.329109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:44.329983 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:44.329986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:44.330799 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:44.330807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:44.331389 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:44.331394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:44.331471 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:44.331551 (+ 80us) client_negotiation.cc:769] Sending connection context
0923 14:36:44.331610 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 14:36:44.331697 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":52,"thread_start_us":53,"threads_started":1}
I20260923 14:36:44.331790 10353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.328235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.2.218.61:45373)
0923 14:36:44.328701 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:44.328707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:44.328720 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:36:44.328763 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:44.328765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:44.328799 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:44.328872 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:44.329223 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:44.329889 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:44.330910 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:44.331296 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:44.331366 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:44.331407 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:44.331641 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 14:36:44.331690 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":50,"threads_started":1}
I20260923 14:36:44.332240 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b007bc31c2df4885ada07189c37f05ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "458bac8acab4452e921aa9185b006045" is_pre_election: true
I20260923 14:36:44.332381 9645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "458bac8acab4452e921aa9185b006045" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b007bc31c2df4885ada07189c37f05ce" is_pre_election: true
I20260923 14:36:44.332412 9833 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b007bc31c2df4885ada07189c37f05ce in term 1.
I20260923 14:36:44.332517 9645 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 458bac8acab4452e921aa9185b006045 in term 1.
I20260923 14:36:44.332731 9620 leader_election.cc:302] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 458bac8acab4452e921aa9185b006045, b007bc31c2df4885ada07189c37f05ce; no voters: d24f06a6135f49f2bdaf29e7f60bb586
I20260923 14:36:44.332830 9810 leader_election.cc:302] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 458bac8acab4452e921aa9185b006045, b007bc31c2df4885ada07189c37f05ce; no voters: d24f06a6135f49f2bdaf29e7f60bb586
I20260923 14:36:44.332870 10346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 14:36:44.332919 10346 raft_consensus.cc:482] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Starting leader election (detected failure of leader d24f06a6135f49f2bdaf29e7f60bb586)
I20260923 14:36:44.332966 10346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:36:44.332980 10347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 14:36:44.333051 10347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d24f06a6135f49f2bdaf29e7f60bb586)
I20260923 14:36:44.333099 10347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:36:44.334131 10346 raft_consensus.cc:504] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } }
I20260923 14:36:44.334153 10347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } }
I20260923 14:36:44.334411 10347 leader_election.cc:288] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 2 election: Requested vote from peers d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787), b007bc31c2df4885ada07189c37f05ce (127.2.218.61:45373)
I20260923 14:36:44.334411 10346 leader_election.cc:288] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [CANDIDATE]: Term 2 election: Requested vote from peers d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787), 458bac8acab4452e921aa9185b006045 (127.2.218.60:33311)
I20260923 14:36:44.334894 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.334672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:35266)
0923 14:36:44.334746 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.334842 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260923 14:36:44.334864 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b007bc31c2df4885ada07189c37f05ce" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "458bac8acab4452e921aa9185b006045"
I20260923 14:36:44.335088 9833 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b007bc31c2df4885ada07189c37f05ce in current term 2: Already voted for candidate 458bac8acab4452e921aa9185b006045 in this term.
I20260923 14:36:44.335135 9645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "458bac8acab4452e921aa9185b006045" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b007bc31c2df4885ada07189c37f05ce"
I20260923 14:36:44.335259 9645 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 458bac8acab4452e921aa9185b006045 in current term 2: Already voted for candidate b007bc31c2df4885ada07189c37f05ce in this term.
I20260923 14:36:44.335376 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.334589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:35262)
0923 14:36:44.335278 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.335322 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":628}
I20260923 14:36:44.335812 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.335660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:35278)
0923 14:36:44.335727 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.335764 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260923 14:36:44.335932 9807 leader_election.cc:334] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787): Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:44.336007 9807 leader_election.cc:302] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 458bac8acab4452e921aa9185b006045; no voters: b007bc31c2df4885ada07189c37f05ce, d24f06a6135f49f2bdaf29e7f60bb586
I20260923 14:36:44.336112 10347 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 14:36:44.336418 10349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.336119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:35290)
0923 14:36:44.336302 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.336371 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260923 14:36:44.336510 9619 leader_election.cc:334] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787): Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:44.336563 9619 leader_election.cc:302] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b007bc31c2df4885ada07189c37f05ce; no voters: 458bac8acab4452e921aa9185b006045, d24f06a6135f49f2bdaf29e7f60bb586
I20260923 14:36:44.336676 10346 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 14:36:44.820101 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.819752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.2:40835)
0923 14:36:44.819976 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.820020 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":93,"threads_started":1}
I20260923 14:36:44.822067 10357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.821642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.1:35125)
0923 14:36:44.821870 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.821954 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260923 14:36:44.822997 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:44.822589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.3:35925)
0923 14:36:44.822830 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:44.822902 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":106,"threads_started":1}
I20260923 14:36:45.821245 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.820780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.2:52251)
0923 14:36:45.821048 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.821095 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":100,"threads_started":1}
I20260923 14:36:45.823001 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.822649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.1:37031)
0923 14:36:45.822865 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.822922 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260923 14:36:45.823879 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.823560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.3:33957)
0923 14:36:45.823733 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.823799 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":48,"threads_started":1}
I20260923 14:36:45.871098 10362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:45.871205 10362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } }
I20260923 14:36:45.871542 10362 leader_election.cc:288] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787), b007bc31c2df4885ada07189c37f05ce (127.2.218.61:45373)
I20260923 14:36:45.871870 9645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "458bac8acab4452e921aa9185b006045" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b007bc31c2df4885ada07189c37f05ce" is_pre_election: true
I20260923 14:36:45.871966 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.871641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:54814)
0923 14:36:45.871836 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.871889 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":56,"threads_started":1}
I20260923 14:36:45.871999 9645 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 458bac8acab4452e921aa9185b006045 in term 2.
I20260923 14:36:45.872262 9810 leader_election.cc:302] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458bac8acab4452e921aa9185b006045, b007bc31c2df4885ada07189c37f05ce; no voters:
I20260923 14:36:45.872406 10362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260923 14:36:45.872471 10362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:45.872517 10362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 2 FOLLOWER]: Advancing to term 3
I20260923 14:36:45.872727 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.872542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:54826)
0923 14:36:45.872632 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.872674 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 14:36:45.872820 9807 leader_election.cc:334] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787): Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:45.873557 10362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } }
I20260923 14:36:45.873854 10362 leader_election.cc:288] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 3 election: Requested vote from peers d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787), b007bc31c2df4885ada07189c37f05ce (127.2.218.61:45373)
I20260923 14:36:45.874151 9645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "458bac8acab4452e921aa9185b006045" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b007bc31c2df4885ada07189c37f05ce"
I20260923 14:36:45.874213 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.873971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:54838)
0923 14:36:45.874049 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.874151 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 14:36:45.874280 9645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 2 FOLLOWER]: Advancing to term 3
I20260923 14:36:45.874795 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.874595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:54844)
0923 14:36:45.874697 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.874741 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 14:36:45.874889 9807 leader_election.cc:334] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787): Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
I20260923 14:36:45.875185 9645 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 458bac8acab4452e921aa9185b006045 in term 3.
I20260923 14:36:45.875463 9810 leader_election.cc:302] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 458bac8acab4452e921aa9185b006045, b007bc31c2df4885ada07189c37f05ce; no voters: d24f06a6135f49f2bdaf29e7f60bb586
I20260923 14:36:45.875624 10362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 14:36:45.875789 10362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [term 3 LEADER]: Becoming Leader. State: Replica: 458bac8acab4452e921aa9185b006045, State: Running, Role: LEADER
I20260923 14:36:45.876037 10362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } }
I20260923 14:36:45.877110 10365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 458bac8acab4452e921aa9185b006045. Latest consensus state: current_term: 3 leader_uuid: "458bac8acab4452e921aa9185b006045" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:45.877189 10365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:45.877427 10367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:45.879536 10367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:45.879894 10367 catalog_manager.cc:1259] Loaded cluster ID: 7a542e1571b642c9817a2e9f29ac9915
I20260923 14:36:45.879959 10367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:45.880424 10367 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:45.880772 10367 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Loaded TSK: 0
I20260923 14:36:45.881388 10367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:36:45.944792 9815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:45.947443 9815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:45.949653 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.949471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:54854)
0923 14:36:45.949565 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.949606 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 14:36:45.949694 9645 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [term 3 FOLLOWER]: Refusing update from remote peer 458bac8acab4452e921aa9185b006045: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260923 14:36:45.950121 10365 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 14:36:45.950366 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.950140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.0.0.1:54856)
0923 14:36:45.950263 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.950312 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 14:36:45.950465 9807 consensus_peers.cc:487] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 -> Peer d24f06a6135f49f2bdaf29e7f60bb586 (127.2.218.62:39787): Couldn't send request to peer d24f06a6135f49f2bdaf29e7f60bb586. Status: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 14:36:45.951396 10369 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 458bac8acab4452e921aa9185b006045. Latest consensus state: current_term: 3 leader_uuid: "458bac8acab4452e921aa9185b006045" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:45.951507 10369 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:36:45.952651 10365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "458bac8acab4452e921aa9185b006045" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:45.952756 10365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:45.953007 10369 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "458bac8acab4452e921aa9185b006045" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d24f06a6135f49f2bdaf29e7f60bb586" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39787 } } peers { permanent_uuid: "b007bc31c2df4885ada07189c37f05ce" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 45373 } attrs { promote: false } } peers { permanent_uuid: "458bac8acab4452e921aa9185b006045" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 33311 } attrs { promote: false } } }
I20260923 14:36:45.953125 10369 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007bc31c2df4885ada07189c37f05ce [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:36:45.961038 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.957319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:37507 (local address 127.0.0.1:43148)
0923 14:36:45.957432 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.957441 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:45.957519 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:45.957818 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:45.957821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:45.957834 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:45.957912 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:45.957917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.958886 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.958889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:45.959966 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:45.959975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.960710 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.960713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:45.960789 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:45.960809 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:36:45.960884 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 14:36:45.960941 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":59}
I20260923 14:36:45.961099 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.957413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.2.218.3:37507)
0923 14:36:45.957633 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:45.957637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:45.957659 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:36:45.957695 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:45.957697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:45.957729 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:45.957811 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:45.958124 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.958781 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.960074 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.960596 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.960719 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:45.960775 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:45.960897 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 14:36:45.960973 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260923 14:36:45.961596 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.957995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:39005 (local address 127.0.0.1:35948)
0923 14:36:45.958591 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.958603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:45.958698 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:45.958896 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:45.958899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:45.958910 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:45.958978 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:45.958982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.959603 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.959607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:45.960428 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:45.960435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.961265 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.961268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:45.961341 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:45.961364 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 14:36:45.961426 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 14:36:45.961484 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":23,"spinlock_wait_cycles":1664,"thread_start_us":65,"threads_started":1}
I20260923 14:36:45.961772 10209 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3cb03840834dd2b8ee10be7c8b3910 (DEFAULT_TABLE table=test-table [id=3b3f6130e56b48ce98a1994e3eb6471e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:45.961936 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.958063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35948 (local address 127.2.218.1:39005)
0923 14:36:45.958435 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:45.958439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:45.958676 (+ 237us) server_negotiation.cc:407] Connection header received
0923 14:36:45.958767 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:45.958772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:45.958824 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:45.958891 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:45.959063 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.959506 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.960526 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.961051 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.961144 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:45.961217 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:45.961733 (+ 516us) server_negotiation.cc:299] Negotiation successful
0923 14:36:45.961806 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":72,"threads_started":1}
I20260923 14:36:45.962028 10209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3cb03840834dd2b8ee10be7c8b3910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:45.962414 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.957752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:46199 (local address 127.0.0.1:51100)
0923 14:36:45.958246 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.958257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:45.958334 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:45.958630 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:45.958632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:45.958664 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:45.958748 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:45.958753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.959639 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.959642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:45.961314 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:45.961323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.962038 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.962042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:45.962144 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:45.962170 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 14:36:45.962254 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 14:36:45.962314 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":33,"spinlock_wait_cycles":768,"thread_start_us":64,"threads_started":1}
I20260923 14:36:45.962478 9936 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3cb03840834dd2b8ee10be7c8b3910 (DEFAULT_TABLE table=test-table [id=3b3f6130e56b48ce98a1994e3eb6471e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:45.962702 9936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3cb03840834dd2b8ee10be7c8b3910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:45.963862 10380 tablet_bootstrap.cc:492] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f: Bootstrap starting.
I20260923 14:36:45.964582 10381 tablet_bootstrap.cc:492] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355: Bootstrap starting.
I20260923 14:36:45.964686 10380 tablet_bootstrap.cc:654] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:45.964771 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.957832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.2.218.2:46199)
0923 14:36:45.958294 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:45.958299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:45.958311 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:45.958438 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:45.958440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:45.958483 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:45.958535 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:45.958835 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.959522 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.961422 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:45.961919 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:45.962017 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:45.962075 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:45.964554 (+ 2479us) server_negotiation.cc:299] Negotiation successful
0923 14:36:45.964652 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":56,"threads_started":1}
I20260923 14:36:45.965183 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 3f3cb03840834dd2b8ee10be7c8b3910 (DEFAULT_TABLE table=test-table [id=3b3f6130e56b48ce98a1994e3eb6471e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:45.965412 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f3cb03840834dd2b8ee10be7c8b3910. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:45.965415 10381 tablet_bootstrap.cc:654] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:45.965687 10380 tablet_bootstrap.cc:492] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f: No bootstrap required, opened a new log
I20260923 14:36:45.965755 10380 ts_tablet_manager.cc:1397] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:45.966085 10380 raft_consensus.cc:348] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } }
I20260923 14:36:45.966228 10380 raft_consensus.cc:374] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:45.966271 10380 raft_consensus.cc:729] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68f17b782494aed8dcec316efb7a76f, State: Initialized, Role: FOLLOWER
I20260923 14:36:45.966383 10380 consensus_queue.cc:260] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } }
I20260923 14:36:45.966449 10381 tablet_bootstrap.cc:492] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355: No bootstrap required, opened a new log
I20260923 14:36:45.966513 10381 ts_tablet_manager.cc:1397] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:45.966701 10380 ts_tablet_manager.cc:1428] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:45.966866 10381 raft_consensus.cc:348] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } }
I20260923 14:36:45.967780 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.967056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.3:49841)
0923 14:36:45.967683 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.967729 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570}
I20260923 14:36:45.967813 10381 raft_consensus.cc:374] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:45.967867 10381 raft_consensus.cc:729] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50c6accd5b564cc5a4cccba961f80355, State: Initialized, Role: FOLLOWER
I20260923 14:36:45.967978 10381 consensus_queue.cc:260] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } }
I20260923 14:36:45.968324 10381 ts_tablet_manager.cc:1428] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260923 14:36:45.968936 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.968711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.1:45627)
0923 14:36:45.968826 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.968875 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 14:36:45.969610 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.969301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.1:54113)
0923 14:36:45.969506 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.969563 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260923 14:36:45.969883 10003 heartbeater.cc:503] Master 127.2.218.60:33311 was elected leader, sending a full tablet report...
I20260923 14:36:45.969934 10385 tablet_bootstrap.cc:492] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7: Bootstrap starting.
I20260923 14:36:45.970748 10385 tablet_bootstrap.cc:654] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:45.971422 10276 heartbeater.cc:503] Master 127.2.218.60:33311 was elected leader, sending a full tablet report...
I20260923 14:36:45.971704 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.971505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.3:49511)
0923 14:36:45.971584 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.971651 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 14:36:45.971943 10385 tablet_bootstrap.cc:492] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7: No bootstrap required, opened a new log
I20260923 14:36:45.972018 10385 ts_tablet_manager.cc:1397] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:45.972425 10385 raft_consensus.cc:348] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } }
I20260923 14:36:45.972570 10385 raft_consensus.cc:374] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:45.972641 10385 raft_consensus.cc:729] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b476e81a19fc420bb11e0666879be1e7, State: Initialized, Role: FOLLOWER
I20260923 14:36:45.972760 10385 consensus_queue.cc:260] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } }
I20260923 14:36:45.973047 10385 ts_tablet_manager.cc:1428] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:45.973577 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.973331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.2:51229)
0923 14:36:45.973434 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.973484 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 14:36:45.974299 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:45.973997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.2:36583)
0923 14:36:45.974193 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:45.974239 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 14:36:45.974711 10140 heartbeater.cc:503] Master 127.2.218.60:33311 was elected leader, sending a full tablet report...
I20260923 14:36:46.008073 10386 raft_consensus.cc:482] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:46.008174 10386 raft_consensus.cc:504] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } }
I20260923 14:36:46.008471 10386 leader_election.cc:288] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c68f17b782494aed8dcec316efb7a76f (127.2.218.3:37507), b476e81a19fc420bb11e0666879be1e7 (127.2.218.2:46199)
I20260923 14:36:46.008765 10229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3cb03840834dd2b8ee10be7c8b3910" candidate_uuid: "50c6accd5b564cc5a4cccba961f80355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68f17b782494aed8dcec316efb7a76f" is_pre_election: true
I20260923 14:36:46.008904 10229 raft_consensus.cc:2454] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50c6accd5b564cc5a4cccba961f80355 in term 0.
I20260923 14:36:46.008867 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3cb03840834dd2b8ee10be7c8b3910" candidate_uuid: "50c6accd5b564cc5a4cccba961f80355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b476e81a19fc420bb11e0666879be1e7" is_pre_election: true
I20260923 14:36:46.009016 10093 raft_consensus.cc:2454] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50c6accd5b564cc5a4cccba961f80355 in term 0.
I20260923 14:36:46.009191 9890 leader_election.cc:302] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50c6accd5b564cc5a4cccba961f80355, c68f17b782494aed8dcec316efb7a76f; no voters:
I20260923 14:36:46.009346 10386 raft_consensus.cc:2781] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:46.009400 10386 raft_consensus.cc:482] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:46.009435 10386 raft_consensus.cc:3037] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:46.010246 10386 raft_consensus.cc:504] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } }
I20260923 14:36:46.010514 10386 leader_election.cc:288] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [CANDIDATE]: Term 1 election: Requested vote from peers c68f17b782494aed8dcec316efb7a76f (127.2.218.3:37507), b476e81a19fc420bb11e0666879be1e7 (127.2.218.2:46199)
I20260923 14:36:46.010735 10229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3cb03840834dd2b8ee10be7c8b3910" candidate_uuid: "50c6accd5b564cc5a4cccba961f80355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68f17b782494aed8dcec316efb7a76f"
I20260923 14:36:46.010854 10229 raft_consensus.cc:3037] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:46.010838 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f3cb03840834dd2b8ee10be7c8b3910" candidate_uuid: "50c6accd5b564cc5a4cccba961f80355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b476e81a19fc420bb11e0666879be1e7"
I20260923 14:36:46.010983 10093 raft_consensus.cc:3037] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:46.011636 10229 raft_consensus.cc:2454] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50c6accd5b564cc5a4cccba961f80355 in term 1.
I20260923 14:36:46.011696 10093 raft_consensus.cc:2454] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50c6accd5b564cc5a4cccba961f80355 in term 1.
I20260923 14:36:46.011881 9890 leader_election.cc:302] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50c6accd5b564cc5a4cccba961f80355, c68f17b782494aed8dcec316efb7a76f; no voters:
I20260923 14:36:46.012019 10386 raft_consensus.cc:2781] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:46.012220 10386 raft_consensus.cc:686] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 1 LEADER]: Becoming Leader. State: Replica: 50c6accd5b564cc5a4cccba961f80355, State: Running, Role: LEADER
I20260923 14:36:46.012353 10386 consensus_queue.cc:237] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } }
I20260923 14:36:46.012774 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:46.012566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.1:42527)
0923 14:36:46.012681 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:46.012728 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 14:36:46.013669 9815 catalog_manager.cc:5495] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50c6accd5b564cc5a4cccba961f80355 (127.2.218.1). New cstate: current_term: 1 leader_uuid: "50c6accd5b564cc5a4cccba961f80355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50c6accd5b564cc5a4cccba961f80355" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 39005 } health_report { overall_health: HEALTHY } } }
I20260923 14:36:46.038940 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:46.035469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.2.218.1:39005)
0923 14:36:46.035608 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:46.035613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:46.035629 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:36:46.035688 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:46.035692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:46.035743 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:46.035823 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:46.036138 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:46.036877 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:46.037936 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:46.038248 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:46.038364 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:46.038585 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:46.038658 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:46.038721 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:46.038797 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 14:36:46.038840 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 14:36:46.040881 10093 raft_consensus.cc:1261] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7 [term 1 FOLLOWER]: Refusing update from remote peer 50c6accd5b564cc5a4cccba961f80355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:46.041368 10386 consensus_queue.cc:1035] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b476e81a19fc420bb11e0666879be1e7" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:46.043349 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:46.043147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.2:40527)
0923 14:36:46.043245 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:46.043289 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 14:36:46.045274 10229 raft_consensus.cc:1261] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f [term 1 FOLLOWER]: Refusing update from remote peer 50c6accd5b564cc5a4cccba961f80355: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:46.045378 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:46.045186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.1:32989)
0923 14:36:46.045269 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:46.045315 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 14:36:46.045697 10386 consensus_queue.cc:1035] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68f17b782494aed8dcec316efb7a76f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:46.046464 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:46.046150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.2:38629)
0923 14:36:46.046379 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:46.046416 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260923 14:36:46.048687 9814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:46.048784 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:46.048400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.3:36595)
0923 14:36:46.048682 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:46.048728 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223}
I20260923 14:36:46.048802 9814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:46.049716 10361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:46.049272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39787 (local address 127.2.218.3:48855)
0923 14:36:46.049593 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:46.049653 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:39787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260923 14:36:46.052464 9814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 458bac8acab4452e921aa9185b006045: Sending DeleteTablet for 3 replicas of tablet 3f3cb03840834dd2b8ee10be7c8b3910
I20260923 14:36:46.053115 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3cb03840834dd2b8ee10be7c8b3910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:46 UTC) from {username='oryx', principal='oryx/127.2.218.60@KRBTEST.COM'} at 127.0.0.1:51100
I20260923 14:36:46.053267 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3cb03840834dd2b8ee10be7c8b3910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:46 UTC) from {username='oryx', principal='oryx/127.2.218.60@KRBTEST.COM'} at 127.0.0.1:51100
I20260923 14:36:46.053398 9936 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3cb03840834dd2b8ee10be7c8b3910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:46 UTC) from {username='oryx', principal='oryx/127.2.218.60@KRBTEST.COM'} at 127.0.0.1:35948
I20260923 14:36:46.053413 10401 tablet_replica.cc:330] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7: stopping tablet replica
I20260923 14:36:46.053519 10401 raft_consensus.cc:2229] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:46.053658 10401 raft_consensus.cc:2258] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:46.053678 10402 tablet_replica.cc:330] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355: stopping tablet replica
I20260923 14:36:46.053815 10402 raft_consensus.cc:2229] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:46.053967 10402 raft_consensus.cc:2258] T 3f3cb03840834dd2b8ee10be7c8b3910 P 50c6accd5b564cc5a4cccba961f80355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:46.054322 10209 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3cb03840834dd2b8ee10be7c8b3910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:46 UTC) from {username='oryx', principal='oryx/127.2.218.60@KRBTEST.COM'} at 127.0.0.1:43148
I20260923 14:36:46.054462 10208 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f3cb03840834dd2b8ee10be7c8b3910 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:46 UTC) from {username='oryx', principal='oryx/127.2.218.60@KRBTEST.COM'} at 127.0.0.1:43148
I20260923 14:36:46.054708 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 9783
I20260923 14:36:46.056870 10403 tablet_replica.cc:330] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f: stopping tablet replica
I20260923 14:36:46.057005 10403 raft_consensus.cc:2229] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:46.057157 10403 raft_consensus.cc:2258] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:46.057781 10403 ts_tablet_manager.cc:1905] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:46.057787 10401 ts_tablet_manager.cc:1905] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:46.059939 10401 ts_tablet_manager.cc:1918] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:46.059939 10403 ts_tablet_manager.cc:1918] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:46.060016 10403 log.cc:1196] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/3f3cb03840834dd2b8ee10be7c8b3910
I20260923 14:36:46.060016 10401 log.cc:1196] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/3f3cb03840834dd2b8ee10be7c8b3910
I20260923 14:36:46.060307 10401 ts_tablet_manager.cc:1939] T 3f3cb03840834dd2b8ee10be7c8b3910 P b476e81a19fc420bb11e0666879be1e7: Deleting consensus metadata
I20260923 14:36:46.060362 10403 ts_tablet_manager.cc:1939] T 3f3cb03840834dd2b8ee10be7c8b3910 P c68f17b782494aed8dcec316efb7a76f: Deleting consensus metadata
I20260923 14:36:46.061003 9807 catalog_manager.cc:4841] TS b476e81a19fc420bb11e0666879be1e7 (127.2.218.2:46199): tablet 3f3cb03840834dd2b8ee10be7c8b3910 (table test-table [id=3b3f6130e56b48ce98a1994e3eb6471e]) successfully deleted
W20260923 14:36:46.061240 9807 catalog_manager.cc:4808] TS b476e81a19fc420bb11e0666879be1e7 (127.2.218.2:46199): delete failed for tablet 3f3cb03840834dd2b8ee10be7c8b3910 because the tablet was not found. No further retry: Not found: Tablet not found: 3f3cb03840834dd2b8ee10be7c8b3910
I20260923 14:36:46.061419 9808 catalog_manager.cc:4841] TS c68f17b782494aed8dcec316efb7a76f (127.2.218.3:37507): tablet 3f3cb03840834dd2b8ee10be7c8b3910 (table test-table [id=3b3f6130e56b48ce98a1994e3eb6471e]) successfully deleted
W20260923 14:36:46.061647 9808 catalog_manager.cc:4808] TS c68f17b782494aed8dcec316efb7a76f (127.2.218.3:37507): delete failed for tablet 3f3cb03840834dd2b8ee10be7c8b3910 because the tablet was not found. No further retry: Not found: Tablet not found: 3f3cb03840834dd2b8ee10be7c8b3910
I20260923 14:36:46.062430 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 10010
I20260923 14:36:46.062577 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:46.062376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:39005 (local address 127.0.0.1:35966)
0923 14:36:46.062458 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:46.062493 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 14:36:46.062932 9807 catalog_manager.cc:4584] TS 50c6accd5b564cc5a4cccba961f80355 (127.2.218.1:39005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f3cb03840834dd2b8ee10be7c8b3910: Network error: Client connection negotiation failed: client connection to 127.2.218.1:39005: connect: Connection refused (error 111)
I20260923 14:36:46.067950 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 10146
I20260923 14:36:46.072675 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 9502
I20260923 14:36:46.077607 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 9689
2026-09-23T14:36:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6630 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:46 dist-test-slave-zl47 krb5kdc[10406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:46 dist-test-slave-zl47 krb5kdc[10406](info): set up 2 sockets
Sep 23 14:36:46 dist-test-slave-zl47 krb5kdc[10406](info): commencing operation
krb5kdc: starting...
W20260923 14:36:46.978716 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.880s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.62.keytab.
Entry for principal oryx/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.62.keytab.
I20260923 14:36:47.078733 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:38257
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.62.keytab
--principal=oryx/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:38257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:47.168181 10419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:47.168401 10419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:47.168483 10419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:47.171820 10419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:47.171895 10419 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 14:36:47.171924 10419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:47.171957 10419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:47.171988 10419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:47.172017 10419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:47.176402 10419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:38257
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.62.keytab
--principal=oryx/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:38257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:47.177129 10419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:47.177878 10419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:47.184700 10425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:47.184706 10427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:47.184792 10419 server_base.cc:1034] running on GCE node
W20260923 14:36:47.184706 10424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:47.185429 10419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:47.186379 10419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:47.187562 10419 hybrid_clock.cc:648] HybridClock initialized: now 1790174207187536 us; error 35 us; skew 500 ppm
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:47.189977 10419 init.cc:377] Logged in from keytab as oryx/127.2.218.62@KRBTEST.COM (short username oryx)
I20260923 14:36:47.190665 10419 webserver.cc:460] Webserver started at http://127.2.218.62:35073/ using document root <none> and password file <none>
I20260923 14:36:47.191061 10419 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:47.191110 10419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:47.191242 10419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:47.192746 10419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "b4947cd6bebd4fff91ccc97a7eadf92c"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "35785cb15c50adb4fa039f9c0c8b06d5"
server_key_iv: "e7ba674d4ce9e6ddf230041820bde4d3"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.193163 10419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "b4947cd6bebd4fff91ccc97a7eadf92c"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "35785cb15c50adb4fa039f9c0c8b06d5"
server_key_iv: "e7ba674d4ce9e6ddf230041820bde4d3"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.196029 10419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260923 14:36:47.197820 10434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:47.198928 10419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:47.199028 10419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "b4947cd6bebd4fff91ccc97a7eadf92c"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "35785cb15c50adb4fa039f9c0c8b06d5"
server_key_iv: "e7ba674d4ce9e6ddf230041820bde4d3"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.199112 10419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:47.207170 10419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:47.209301 10419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:47.209415 10419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:47.215539 10419 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:38257
I20260923 14:36:47.215561 10496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:38257 every 8 connection(s)
I20260923 14:36:47.216234 10419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:47.218351 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:47.222644 10497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c: Bootstrap starting.
I20260923 14:36:47.222848 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 10419
I20260923 14:36:47.222965 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:47.223227 2920 external_mini_cluster.cc:1428] Setting key 1f52769b767a879ed029b5b626a12cff
I20260923 14:36:47.224812 10497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:47.225400 10497 log.cc:824] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:47.226902 10497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c: No bootstrap required, opened a new log
I20260923 14:36:47.229735 10497 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4947cd6bebd4fff91ccc97a7eadf92c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38257 } }
I20260923 14:36:47.229934 10497 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:47.230003 10497 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4947cd6bebd4fff91ccc97a7eadf92c, State: Initialized, Role: FOLLOWER
I20260923 14:36:47.230405 10497 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4947cd6bebd4fff91ccc97a7eadf92c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38257 } }
I20260923 14:36:47.230530 10497 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:47.230605 10497 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:47.230685 10497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [term 0 FOLLOWER]: Advancing to term 1
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:47.231412 10497 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4947cd6bebd4fff91ccc97a7eadf92c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38257 } }
I20260923 14:36:47.231760 10497 leader_election.cc:302] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4947cd6bebd4fff91ccc97a7eadf92c; no voters:
I20260923 14:36:47.232067 10497 leader_election.cc:288] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:47.232123 10502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:47.232532 10502 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [term 1 LEADER]: Becoming Leader. State: Replica: b4947cd6bebd4fff91ccc97a7eadf92c, State: Running, Role: LEADER
I20260923 14:36:47.232816 10502 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4947cd6bebd4fff91ccc97a7eadf92c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38257 } }
I20260923 14:36:47.233054 10497 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:47.233757 10502 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4947cd6bebd4fff91ccc97a7eadf92c. Latest consensus state: current_term: 1 leader_uuid: "b4947cd6bebd4fff91ccc97a7eadf92c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4947cd6bebd4fff91ccc97a7eadf92c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38257 } } }
I20260923 14:36:47.233968 10502 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:47.234190 10503 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4947cd6bebd4fff91ccc97a7eadf92c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4947cd6bebd4fff91ccc97a7eadf92c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38257 } } }
I20260923 14:36:47.234361 10503 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:47.234366 10511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:47.236685 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.224311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.2.218.62:38257)
0923 14:36:47.224684 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.224696 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.224725 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:36:47.225280 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.225294 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.225600 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.225877 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:47.226692 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.227572 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.228335 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.228807 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.228975 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.231412 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:47.231418 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:47.231433 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:47.231462 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:47.233188 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.233596 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.233603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.233608 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.233752 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.234216 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.234219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.234221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.234751 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:47.235496 (+ 745us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.235649 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.235906 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260923 14:36:47.237989 10511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:47.245514 10511 catalog_manager.cc:1347] Generated new cluster ID: 6bd8973af5ab40d986b4a93093be4878
I20260923 14:36:47.245606 10511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:47.263103 10511 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:47.264034 10511 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:47.272240 10511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c: Generated new TSK 0
I20260923 14:36:47.272778 10511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab.
Entry for principal oryx/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab.
I20260923 14:36:47.300941 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38257
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab
--principal=oryx/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:47.392874 10523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:47.393070 10523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:47.393105 10523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:47.396723 10523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:47.396764 10523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:47.396826 10523 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:36:47.396894 10523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:47.401196 10523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab
--principal=oryx/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.218.62:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:47.401943 10523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:47.402781 10523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:47.410530 10531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:47.410521 10528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:47.410678 10523 server_base.cc:1034] running on GCE node
W20260923 14:36:47.410521 10529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:47.411167 10523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:47.411660 10523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:47.412796 10523 hybrid_clock.cc:648] HybridClock initialized: now 1790174207412774 us; error 33 us; skew 500 ppm
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:47.415057 10523 init.cc:377] Logged in from keytab as oryx/127.2.218.1@KRBTEST.COM (short username oryx)
I20260923 14:36:47.415731 10523 webserver.cc:460] Webserver started at http://127.2.218.1:36291/ using document root <none> and password file <none>
I20260923 14:36:47.416175 10523 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:47.416252 10523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:47.416463 10523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:47.418054 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "4bb5d88476fc45bcb840ef1862740ce5"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "da27c73dc78f91a2b35aafb71d86d462"
server_key_iv: "a41d6e419b49afe9a9c2dcdacff73e0e"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.418573 10523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "4bb5d88476fc45bcb840ef1862740ce5"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "da27c73dc78f91a2b35aafb71d86d462"
server_key_iv: "a41d6e419b49afe9a9c2dcdacff73e0e"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.421558 10523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:36:47.423537 10538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:47.424461 10523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:47.424573 10523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "4bb5d88476fc45bcb840ef1862740ce5"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "da27c73dc78f91a2b35aafb71d86d462"
server_key_iv: "a41d6e419b49afe9a9c2dcdacff73e0e"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.424669 10523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:47.446943 10523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:47.449196 10523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:47.449353 10523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:47.450632 10523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:47.450723 10523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:47.450793 10523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:47.450832 10523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:47.460170 10523 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:37979
I20260923 14:36:47.460345 10653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:37979 every 8 connection(s)
I20260923 14:36:47.461673 10523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:47.462728 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.451212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:37219 (local address 127.2.218.62:38257)
0923 14:36:47.451343 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.451348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.452089 (+ 741us) server_negotiation.cc:407] Connection header received
0923 14:36:47.453023 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.453029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.453087 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.453156 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:47.454573 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.455126 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.456265 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.456517 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.456609 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.459360 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:47.459363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:47.459366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:47.459391 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:47.460794 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.461309 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.461312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.461313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.461393 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.461695 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.461698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.461700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.461897 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:47.461995 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.462529 (+ 534us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.462616 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:36:47.463407 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.451563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38257 (local address 127.2.218.1:37219)
0923 14:36:47.451941 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:47.451982 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:47.452801 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:47.453545 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:47.453555 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:47.453971 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:47.454445 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.454456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.455244 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.455249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.456147 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.456156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.456660 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.456666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.456860 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.457399 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:47.457422 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:47.459203 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:47.460916 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.460921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.460932 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.461218 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.461476 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.461477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.461479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.461583 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.462003 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:47.462008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:47.462278 (+ 270us) client_negotiation.cc:769] Sending connection context
0923 14:36:47.462467 (+ 189us) client_negotiation.cc:240] Negotiation successful
0923 14:36:47.462689 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
I20260923 14:36:47.464555 10654 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38257
I20260923 14:36:47.464684 10654 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:47.465083 10654 heartbeater.cc:511] Master 127.2.218.62:38257 requested a full tablet report, sending...
I20260923 14:36:47.465677 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 10523
I20260923 14:36:47.465794 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:47.466084 2920 external_mini_cluster.cc:1428] Setting key f00ded17eda5bb889970859d37acfe48
I20260923 14:36:47.466452 10450 ts_manager.cc:194] Registered new tserver with Master: 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979)
I20260923 14:36:47.467545 10450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:37219
I20260923 14:36:47.475028 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.466455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38257 (local address 127.2.218.1:52213)
0923 14:36:47.466746 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:47.466759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:47.466840 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:47.468242 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:47.468247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:47.468448 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:47.468634 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.468644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.469530 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.469535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.470431 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.470450 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.470920 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.470926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.471055 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.471541 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:47.471551 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:47.471831 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:47.473691 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.473693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.473694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.473852 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.474172 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.474174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.474175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.474240 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.474662 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:47.474664 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:47.474729 (+ 65us) client_negotiation.cc:769] Sending connection context
0923 14:36:47.474809 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:36:47.474895 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":3,"thread_start_us":119,"threads_started":1}
I20260923 14:36:47.475217 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.466531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:52213 (local address 127.2.218.62:38257)
0923 14:36:47.468027 (+ 1496us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.468030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.468062 (+ 32us) server_negotiation.cc:407] Connection header received
0923 14:36:47.468091 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.468093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.468127 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.468203 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:47.468753 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.469396 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.470565 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.470801 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.470884 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.472026 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:47.472032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:47.472036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:47.472065 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:47.473591 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.473943 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.473947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.473949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.474031 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.474327 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.474332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.474335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.474553 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:47.474654 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.474891 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.475024 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1448}
WARNING: no policy specified for oryx/127.2.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab.
Entry for principal oryx/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab.
I20260923 14:36:47.490852 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38257
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab
--principal=oryx/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:47.585489 10663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:47.585681 10663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:47.585716 10663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:47.589133 10663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:47.589176 10663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:47.589239 10663 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:36:47.589311 10663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:47.593623 10663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab
--principal=oryx/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.218.62:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:47.594504 10663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:47.595355 10663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:47.603137 10671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:47.603255 10668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:47.603124 10669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:47.603453 10663 server_base.cc:1034] running on GCE node
I20260923 14:36:47.603869 10663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:47.604388 10663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:47.605566 10663 hybrid_clock.cc:648] HybridClock initialized: now 1790174207605532 us; error 40 us; skew 500 ppm
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:47.608150 10663 init.cc:377] Logged in from keytab as oryx/127.2.218.2@KRBTEST.COM (short username oryx)
I20260923 14:36:47.608901 10663 webserver.cc:460] Webserver started at http://127.2.218.2:33943/ using document root <none> and password file <none>
I20260923 14:36:47.609382 10663 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:47.609469 10663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:47.609685 10663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:47.611637 10663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "71edd6118fe546aeabda4457aef8987f"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "dfca952559f501948ff58f2dec32a10f"
server_key_iv: "ab13a5f07f0a8800efc04272551ed388"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.612203 10663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "71edd6118fe546aeabda4457aef8987f"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "dfca952559f501948ff58f2dec32a10f"
server_key_iv: "ab13a5f07f0a8800efc04272551ed388"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.615505 10663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:36:47.617463 10678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:47.618495 10663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:36:47.618620 10663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "71edd6118fe546aeabda4457aef8987f"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "dfca952559f501948ff58f2dec32a10f"
server_key_iv: "ab13a5f07f0a8800efc04272551ed388"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.618716 10663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:47.645736 10663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:47.648245 10663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:47.648432 10663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:47.649777 10663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:47.649847 10663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:47.649912 10663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:47.649956 10663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:47.659842 10663 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:46797
I20260923 14:36:47.659885 10793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:46797 every 8 connection(s)
I20260923 14:36:47.660742 10663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:36:47.662367 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.650509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:36033 (local address 127.2.218.62:38257)
0923 14:36:47.650661 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.650666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.651316 (+ 650us) server_negotiation.cc:407] Connection header received
0923 14:36:47.652209 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.652213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.652266 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.652370 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:47.653631 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.654420 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.655554 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.655878 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.655981 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.659095 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:47.659099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:47.659102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:47.659136 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:47.660509 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.661064 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.661067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.661068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.661192 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.661540 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.661542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.661543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.661686 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:47.661751 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.662182 (+ 431us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.662267 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:36:47.663090 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.650797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38257 (local address 127.2.218.2:36033)
0923 14:36:47.651153 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:47.651195 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:47.651992 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:47.652517 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:47.652524 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:47.652946 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:47.653466 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.653484 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.654539 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.654542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.655446 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.655454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.656018 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.656022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.656197 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.656776 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:47.656795 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:47.658945 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:47.660621 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.660626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.660637 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.660967 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.661294 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.661297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.661298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.661440 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.661770 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:47.661774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:47.661947 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 14:36:47.662086 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 14:36:47.662366 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
I20260923 14:36:47.664165 10794 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38257
I20260923 14:36:47.664316 10794 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:47.664769 10794 heartbeater.cc:511] Master 127.2.218.62:38257 requested a full tablet report, sending...
I20260923 14:36:47.665740 10450 ts_manager.cc:194] Registered new tserver with Master: 71edd6118fe546aeabda4457aef8987f (127.2.218.2:46797)
I20260923 14:36:47.665856 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 10663
I20260923 14:36:47.665977 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:47.666272 2920 external_mini_cluster.cc:1428] Setting key f5e0bf0f73df2bbea5dfa507c6188b25
I20260923 14:36:47.666373 10450 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:36033
I20260923 14:36:47.674103 10800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.665945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38257 (local address 127.2.218.2:48921)
0923 14:36:47.666663 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:47.666677 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:47.666777 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:47.666985 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:47.666987 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:47.667182 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:47.667288 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.667294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.668983 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.668985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.669459 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.669464 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.669956 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.669959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.670028 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.670526 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:47.670539 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:47.670845 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:47.672706 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.672708 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.672710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.672887 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.673178 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.673180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.673181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.673251 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.673723 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:47.673725 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:47.673791 (+ 66us) client_negotiation.cc:769] Sending connection context
0923 14:36:47.673873 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 14:36:47.673965 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":99,"threads_started":1}
I20260923 14:36:47.674256 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.666003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:48921 (local address 127.2.218.62:38257)
0923 14:36:47.666090 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.666114 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.666746 (+ 632us) server_negotiation.cc:407] Connection header received
0923 14:36:47.666860 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.666863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.666896 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.666972 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:47.667392 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.668000 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.669562 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.669852 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.669948 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.671030 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:47.671037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:47.671042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:47.671071 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:47.672594 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.672989 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.672993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.672995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.673092 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.673339 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.673343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.673346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.673616 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:47.673729 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.673951 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.674067 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for oryx/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab.
Entry for principal oryx/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab.
I20260923 14:36:47.690536 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38257
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab
--principal=oryx/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:47.785169 10803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:47.785384 10803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:47.785450 10803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:47.788952 10803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:47.789027 10803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:47.789090 10803 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 14:36:47.789165 10803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:47.793560 10803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab
--principal=oryx/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.218.62:38257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:47.794374 10803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:47.795202 10803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:47.803326 10809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:47.803354 10808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:47.803491 10811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:47.803368 10803 server_base.cc:1034] running on GCE node
I20260923 14:36:47.803874 10803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:47.804472 10803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:47.805635 10803 hybrid_clock.cc:648] HybridClock initialized: now 1790174207805622 us; error 23 us; skew 500 ppm
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:47.808058 10803 init.cc:377] Logged in from keytab as oryx/127.2.218.3@KRBTEST.COM (short username oryx)
I20260923 14:36:47.808737 10803 webserver.cc:460] Webserver started at http://127.2.218.3:42847/ using document root <none> and password file <none>
I20260923 14:36:47.809202 10803 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:47.809278 10803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:47.809460 10803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:47.811206 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "92697923160649df98e76ba50b1bdf1a"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "71db8f63ccfa484b728d903bfa45d4ad"
server_key_iv: "2aaf1f37b3e452529f71429865ee3cb9"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.811679 10803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "92697923160649df98e76ba50b1bdf1a"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "71db8f63ccfa484b728d903bfa45d4ad"
server_key_iv: "2aaf1f37b3e452529f71429865ee3cb9"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.814711 10803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:36:47.816680 10818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:47.817576 10803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:36:47.817713 10803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "92697923160649df98e76ba50b1bdf1a"
format_stamp: "Formatted at 2026-09-23 14:36:47 on dist-test-slave-zl47"
server_key: "71db8f63ccfa484b728d903bfa45d4ad"
server_key_iv: "2aaf1f37b3e452529f71429865ee3cb9"
server_key_version: "encryptionkey@0"
I20260923 14:36:47.817806 10803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:47.831527 10803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:47.834378 10803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:47.834539 10803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:47.835683 10803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:47.835760 10803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:47.835819 10803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:47.835876 10803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:47.845402 10803 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:44657
I20260923 14:36:47.845471 10933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:44657 every 8 connection(s)
I20260923 14:36:47.846150 10803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:36:47.848276 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.836402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:37729 (local address 127.2.218.62:38257)
0923 14:36:47.836510 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.836513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.837335 (+ 822us) server_negotiation.cc:407] Connection header received
0923 14:36:47.838268 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.838274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.838335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.838390 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:47.839916 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.840673 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.842322 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.842653 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.842751 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.845296 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:47.845299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:47.845301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:47.845323 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:47.846877 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.847280 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.847283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.847284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.847351 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.847568 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.847570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.847571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.847722 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:47.847784 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.848092 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.848177 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:36:47.848887 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.836807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38257 (local address 127.2.218.3:37729)
0923 14:36:47.837202 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:47.837236 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:47.838001 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:47.838771 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:47.838780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:47.839210 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:47.839744 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.839772 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.840791 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.840794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.842188 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.842199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.842799 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.842803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.842986 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.843455 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:47.843476 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:47.845157 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:47.846971 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.846974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.846983 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.847198 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.847406 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.847408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.847409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.847503 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.847777 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:47.847780 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:47.847935 (+ 155us) client_negotiation.cc:769] Sending connection context
0923 14:36:47.848053 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 14:36:47.848229 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":100,"threads_started":1}
I20260923 14:36:47.849829 10934 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38257
I20260923 14:36:47.849946 10934 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:47.850507 10934 heartbeater.cc:511] Master 127.2.218.62:38257 requested a full tablet report, sending...
I20260923 14:36:47.851485 10451 ts_manager.cc:194] Registered new tserver with Master: 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657)
I20260923 14:36:47.852155 10451 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:37729
I20260923 14:36:47.854773 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 10803
I20260923 14:36:47.854888 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:47.855194 2920 external_mini_cluster.cc:1428] Setting key 5bf1a549e6d0626158a7ba11d06ffe87
I20260923 14:36:47.857069 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:36:47.857936 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.851896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:56991 (local address 127.2.218.62:38257)
0923 14:36:47.852004 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.852008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.852143 (+ 135us) server_negotiation.cc:407] Connection header received
0923 14:36:47.852233 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.852236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.852273 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.852351 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:47.852721 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.853110 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.853977 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.854215 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.854282 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.855198 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:47.855201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:47.855203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:47.855220 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:47.856680 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.856983 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.856987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.856989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.857058 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.857318 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.857320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.857320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.857445 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:47.857508 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.857779 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.857850 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:36:47.858106 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.851845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38257 (local address 127.2.218.3:56991)
0923 14:36:47.852056 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:47.852067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:47.852170 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:47.852363 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:47.852366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:47.852521 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:47.852613 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.852621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.853240 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.853243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.853886 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.853894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.854305 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.854307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.854371 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.854694 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:47.854708 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:47.855056 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:47.856768 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.856769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.856771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.856921 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.857162 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:47.857164 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:47.857166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:47.857240 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:47.857527 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:47.857531 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:47.857611 (+ 80us) client_negotiation.cc:769] Sending connection context
0923 14:36:47.857737 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 14:36:47.857871 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 14:36:47.877169 10500 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:47.872245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.2.218.62:38257)
0923 14:36:47.872350 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.872352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.872362 (+ 10us) server_negotiation.cc:407] Connection header received
0923 14:36:47.872388 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.872389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.872416 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.872472 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:47.873036 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.873426 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.874158 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.874345 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.874414 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.877059 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39864: BlockingRecv error: recv got EOF from 127.0.0.1:39864 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 23 14:36:47 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:47.884783 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.878300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.2.218.62:38257)
0923 14:36:47.878400 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.878403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.878510 (+ 107us) server_negotiation.cc:407] Connection header received
0923 14:36:47.878566 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.878568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.878595 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.878650 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:47.879078 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.879467 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.880065 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.880237 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.880299 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.882193 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:47.882196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:47.882197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:47.882217 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:47.883476 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.883824 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.883826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.883827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.883897 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:47.884102 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:47.884104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:47.884126 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:47.884274 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:47.884377 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.884513 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.884587 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:36:47.887308 10451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:47.889905 10451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:47.903438 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.898197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.2.218.1:37979)
0923 14:36:47.898985 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.898994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.899014 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:36:47.899118 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.899124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.899295 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.899425 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:47.899597 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.900262 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.901933 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.902706 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.902838 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.903036 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.903196 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.903299 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":123,"threads_started":1}
I20260923 14:36:47.903443 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.897696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:37979 (local address 127.0.0.1:47880)
0923 14:36:47.898642 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:47.898656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:47.898738 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:47.899422 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:47.899424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:47.899439 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:47.899508 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.899515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.900379 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.900385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.901824 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.901839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.902867 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.902872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.903001 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.903044 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 14:36:47.903151 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 14:36:47.903275 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":157,"thread_start_us":111,"threads_started":1}
I20260923 14:36:47.903638 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.897413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:46797 (local address 127.0.0.1:33234)
0923 14:36:47.897975 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:47.897991 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:47.898155 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:47.898744 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:47.898751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:47.898775 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:47.898880 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.898886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.900352 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.900358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.902150 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.902160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.903246 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.903249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.903334 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.903359 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 14:36:47.903435 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 14:36:47.903510 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":250,"thread_start_us":84,"threads_started":1}
I20260923 14:36:47.903791 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.897380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.2.218.3:44657)
0923 14:36:47.898730 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.898741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.898760 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:36:47.898822 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.898826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.898998 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.899169 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:47.899397 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.900335 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.902070 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.902738 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.902840 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.902987 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.903523 (+ 536us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.903657 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":79,"threads_started":1}
I20260923 14:36:47.904359 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.897785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.2.218.2:46797)
0923 14:36:47.898236 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:47.898250 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:47.898275 (+ 25us) server_negotiation.cc:407] Connection header received
0923 14:36:47.898352 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:47.898358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:47.898560 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:47.899025 (+ 465us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:47.899181 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.900217 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.902247 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.902890 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.903001 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.903182 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:47.904117 (+ 935us) server_negotiation.cc:299] Negotiation successful
0923 14:36:47.904215 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":117,"threads_started":1}
I20260923 14:36:47.903460 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:47.897282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:44657 (local address 127.0.0.1:42366)
0923 14:36:47.897900 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:47.897979 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:47.898487 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:47.899164 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:47.899167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:47.899184 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:47.899284 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.899289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.900465 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.900468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.901944 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:47.901955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:47.903053 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:47.903079 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:47.903180 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:47.903204 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 14:36:47.903271 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 14:36:47.903332 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":119,"thread_start_us":48,"threads_started":1}
I20260923 14:36:47.906404 10588 tablet_service.cc:1467] Processing CreateTablet for tablet 837e8a2179894512b24973eec6115428 (DEFAULT_TABLE table=test-table [id=1b8b0e59cea8469497c5d693fd2d020d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:47.906563 10728 tablet_service.cc:1467] Processing CreateTablet for tablet 837e8a2179894512b24973eec6115428 (DEFAULT_TABLE table=test-table [id=1b8b0e59cea8469497c5d693fd2d020d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:47.907272 10588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e8a2179894512b24973eec6115428. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:47.907421 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e8a2179894512b24973eec6115428. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:47.913887 10962 tablet_bootstrap.cc:492] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: Bootstrap starting.
I20260923 14:36:47.913797 10868 tablet_service.cc:1467] Processing CreateTablet for tablet 837e8a2179894512b24973eec6115428 (DEFAULT_TABLE table=test-table [id=1b8b0e59cea8469497c5d693fd2d020d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:47.914593 10868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837e8a2179894512b24973eec6115428. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:47.915622 10962 tablet_bootstrap.cc:654] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:47.916262 10962 log.cc:824] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:47.917064 10963 tablet_bootstrap.cc:492] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: Bootstrap starting.
I20260923 14:36:47.918031 10962 tablet_bootstrap.cc:492] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: No bootstrap required, opened a new log
I20260923 14:36:47.918228 10962 ts_tablet_manager.cc:1397] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:36:47.918591 10963 tablet_bootstrap.cc:654] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:47.919245 10963 log.cc:824] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:47.920298 10966 tablet_bootstrap.cc:492] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: Bootstrap starting.
I20260923 14:36:47.921516 10963 tablet_bootstrap.cc:492] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: No bootstrap required, opened a new log
I20260923 14:36:47.921681 10963 ts_tablet_manager.cc:1397] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:36:47.921586 10962 raft_consensus.cc:348] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } }
I20260923 14:36:47.921797 10962 raft_consensus.cc:374] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:47.921845 10962 raft_consensus.cc:729] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71edd6118fe546aeabda4457aef8987f, State: Initialized, Role: FOLLOWER
I20260923 14:36:47.922087 10966 tablet_bootstrap.cc:654] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:47.922287 10962 consensus_queue.cc:260] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } }
I20260923 14:36:47.922933 10794 heartbeater.cc:503] Master 127.2.218.62:38257 was elected leader, sending a full tablet report...
I20260923 14:36:47.923280 10962 ts_tablet_manager.cc:1428] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:36:47.923841 10966 log.cc:824] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:47.925237 10966 tablet_bootstrap.cc:492] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: No bootstrap required, opened a new log
I20260923 14:36:47.925382 10966 ts_tablet_manager.cc:1397] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:36:47.926321 10963 raft_consensus.cc:348] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } }
I20260923 14:36:47.926532 10963 raft_consensus.cc:374] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:47.926585 10963 raft_consensus.cc:729] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb5d88476fc45bcb840ef1862740ce5, State: Initialized, Role: FOLLOWER
I20260923 14:36:47.926973 10963 consensus_queue.cc:260] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } }
I20260923 14:36:47.927505 10654 heartbeater.cc:503] Master 127.2.218.62:38257 was elected leader, sending a full tablet report...
I20260923 14:36:47.927788 10963 ts_tablet_manager.cc:1428] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:36:47.928681 10966 raft_consensus.cc:348] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } }
I20260923 14:36:47.928862 10966 raft_consensus.cc:374] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:47.928912 10966 raft_consensus.cc:729] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92697923160649df98e76ba50b1bdf1a, State: Initialized, Role: FOLLOWER
I20260923 14:36:47.929243 10966 consensus_queue.cc:260] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } }
I20260923 14:36:47.929771 10934 heartbeater.cc:503] Master 127.2.218.62:38257 was elected leader, sending a full tablet report...
I20260923 14:36:47.930013 10966 ts_tablet_manager.cc:1428] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260923 14:36:47.962673 10655 tablet.cc:2369] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:48.059806 10967 raft_consensus.cc:482] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:48.060010 10967 raft_consensus.cc:504] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } }
I20260923 14:36:48.061043 10967 leader_election.cc:288] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657), 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979)
I20260923 14:36:48.065377 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.061390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:48489 (local address 127.2.218.1:37979)
0923 14:36:48.061518 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.061522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.061663 (+ 141us) server_negotiation.cc:407] Connection header received
0923 14:36:48.061735 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.061739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.061780 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.061845 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:48.062057 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.062759 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.064266 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.064823 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.064922 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.064994 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.065163 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.065244 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 14:36:48.065388 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.061277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:37979 (local address 127.2.218.2:48489)
0923 14:36:48.061578 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.061590 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.061670 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.061862 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.061864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.061874 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:48.061969 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.061975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.062864 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.062867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.064151 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.064162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.064943 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.064945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.065023 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.065043 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.065152 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.065218 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":53,"thread_start_us":93,"threads_started":1}
I20260923 14:36:48.065505 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.061299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:59623 (local address 127.2.218.3:44657)
0923 14:36:48.061406 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.061410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.061506 (+ 96us) server_negotiation.cc:407] Connection header received
0923 14:36:48.061600 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.061602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.061668 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.061746 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:48.062043 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.062756 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.064438 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.065026 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.065137 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.065190 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.065330 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.065396 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:36:48.065867 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.061260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:44657 (local address 127.2.218.2:59623)
0923 14:36:48.061411 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.061424 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.061510 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.061779 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.061781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.061793 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:48.061922 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.061928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.062864 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.062867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.064330 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.064341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.065133 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.065136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.065216 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.065233 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.065299 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.065351 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":32}
I20260923 14:36:48.066002 10608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e8a2179894512b24973eec6115428" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb5d88476fc45bcb840ef1862740ce5" is_pre_election: true
I20260923 14:36:48.066298 10608 raft_consensus.cc:2454] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 0.
I20260923 14:36:48.066457 10874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e8a2179894512b24973eec6115428" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92697923160649df98e76ba50b1bdf1a" is_pre_election: true
I20260923 14:36:48.066670 10874 raft_consensus.cc:2454] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 0.
I20260923 14:36:48.066751 10681 leader_election.cc:302] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb5d88476fc45bcb840ef1862740ce5, 71edd6118fe546aeabda4457aef8987f; no voters:
I20260923 14:36:48.067021 10967 raft_consensus.cc:2781] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:48.067108 10967 raft_consensus.cc:482] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:48.067147 10967 raft_consensus.cc:3037] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.068065 10967 raft_consensus.cc:504] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } }
I20260923 14:36:48.068351 10967 leader_election.cc:288] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 election: Requested vote from peers 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657), 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979)
I20260923 14:36:48.068590 10874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e8a2179894512b24973eec6115428" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92697923160649df98e76ba50b1bdf1a"
I20260923 14:36:48.068683 10608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837e8a2179894512b24973eec6115428" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb5d88476fc45bcb840ef1862740ce5"
I20260923 14:36:48.068729 10874 raft_consensus.cc:3037] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.068812 10608 raft_consensus.cc:3037] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.069648 10874 raft_consensus.cc:2454] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 1.
I20260923 14:36:48.069741 10608 raft_consensus.cc:2454] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 1.
I20260923 14:36:48.070020 10680 leader_election.cc:302] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71edd6118fe546aeabda4457aef8987f, 92697923160649df98e76ba50b1bdf1a; no voters:
I20260923 14:36:48.070227 10967 raft_consensus.cc:2781] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:48.070386 10967 raft_consensus.cc:686] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 1 LEADER]: Becoming Leader. State: Replica: 71edd6118fe546aeabda4457aef8987f, State: Running, Role: LEADER
I20260923 14:36:48.070655 10967 consensus_queue.cc:237] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } }
I20260923 14:36:48.073354 10451 catalog_manager.cc:5495] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f reported cstate change: term changed from 0 to 1, leader changed from <none> to 71edd6118fe546aeabda4457aef8987f (127.2.218.2). New cstate: current_term: 1 leader_uuid: "71edd6118fe546aeabda4457aef8987f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } health_report { overall_health: UNKNOWN } } }
I20260923 14:36:48.087309 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.084601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.2.218.2:46797)
0923 14:36:48.084717 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.084720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.084731 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:36:48.084858 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.084860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.084899 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.084970 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:48.085157 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.085554 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.086295 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.086466 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.086531 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.086727 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.087051 (+ 324us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.087123 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.087195 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.087235 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:36:48.091826 10608 raft_consensus.cc:1261] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Refusing update from remote peer 71edd6118fe546aeabda4457aef8987f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:48.091832 10874 raft_consensus.cc:1261] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Refusing update from remote peer 71edd6118fe546aeabda4457aef8987f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:48.092348 10972 consensus_queue.cc:1035] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:48.092519 10967 consensus_queue.cc:1035] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 14:36:48.097510 10935 tablet.cc:2369] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:36:48.099493 10976 mvcc.cc:205] Tried to move back new op lower bound from 7332553556338585600 to 7332553556259438592. Current Snapshot: MvccSnapshot[applied={T|T < 7332553556338585600}]
I20260923 14:36:48.106383 10451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:48.106528 10451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:48.108536 10451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c: Sending DeleteTablet for 3 replicas of tablet 837e8a2179894512b24973eec6115428
I20260923 14:36:48.109191 10868 tablet_service.cc:1514] Processing DeleteTablet for tablet 837e8a2179894512b24973eec6115428 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:42366
I20260923 14:36:48.109316 10588 tablet_service.cc:1514] Processing DeleteTablet for tablet 837e8a2179894512b24973eec6115428 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:47880
I20260923 14:36:48.109601 10728 tablet_service.cc:1514] Processing DeleteTablet for tablet 837e8a2179894512b24973eec6115428 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:33234
I20260923 14:36:48.109853 10986 tablet_replica.cc:330] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: stopping tablet replica
I20260923 14:36:48.109953 10451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 14:36:48.110168 10986 raft_consensus.cc:2229] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260923 14:36:48.110333 10451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:48.110416 10986 raft_consensus.cc:2258] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.111037 10985 tablet_replica.cc:330] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: stopping tablet replica
I20260923 14:36:48.111260 10985 raft_consensus.cc:2229] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:48.111774 10985 raft_consensus.cc:2258] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.111980 10986 ts_tablet_manager.cc:1905] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.114149 10588 tablet_service.cc:1467] Processing CreateTablet for tablet 7c595d965f2d46839d2d65cb69b543b8 (DEFAULT_TABLE table=test-table [id=08e4d1517c2e44b3a1b8e05263c88f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:48.114298 10868 tablet_service.cc:1467] Processing CreateTablet for tablet 7c595d965f2d46839d2d65cb69b543b8 (DEFAULT_TABLE table=test-table [id=08e4d1517c2e44b3a1b8e05263c88f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:48.114369 10588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c595d965f2d46839d2d65cb69b543b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.114410 10728 tablet_service.cc:1467] Processing CreateTablet for tablet 7c595d965f2d46839d2d65cb69b543b8 (DEFAULT_TABLE table=test-table [id=08e4d1517c2e44b3a1b8e05263c88f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:48.114507 10868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c595d965f2d46839d2d65cb69b543b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.114609 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c595d965f2d46839d2d65cb69b543b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.116330 10963 tablet_bootstrap.cc:492] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5: Bootstrap starting.
I20260923 14:36:48.116791 10962 tablet_bootstrap.cc:492] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: Bootstrap starting.
I20260923 14:36:48.117174 10963 tablet_bootstrap.cc:654] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.117556 10962 tablet_bootstrap.cc:654] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.117970 10987 tablet_replica.cc:330] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: stopping tablet replica
I20260923 14:36:48.118036 10966 tablet_bootstrap.cc:492] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a: Bootstrap starting.
I20260923 14:36:48.118217 10987 raft_consensus.cc:2229] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:48.118510 10987 raft_consensus.cc:2258] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.119243 10963 tablet_bootstrap.cc:492] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5: No bootstrap required, opened a new log
I20260923 14:36:48.119256 10966 tablet_bootstrap.cc:654] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.119309 10963 ts_tablet_manager.cc:1397] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:36:48.119679 10963 raft_consensus.cc:348] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.119760 10963 raft_consensus.cc:374] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.119786 10963 raft_consensus.cc:729] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb5d88476fc45bcb840ef1862740ce5, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.119886 10963 consensus_queue.cc:260] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.120093 10963 ts_tablet_manager.cc:1428] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 14:36:48.120834 10985 ts_tablet_manager.cc:1905] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.121148 10987 ts_tablet_manager.cc:1905] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.122778 10966 tablet_bootstrap.cc:492] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a: No bootstrap required, opened a new log
I20260923 14:36:48.122850 10966 ts_tablet_manager.cc:1397] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260923 14:36:48.123234 10966 raft_consensus.cc:348] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.123337 10966 raft_consensus.cc:374] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.123380 10966 raft_consensus.cc:729] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92697923160649df98e76ba50b1bdf1a, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.123495 10966 consensus_queue.cc:260] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.123728 10966 ts_tablet_manager.cc:1428] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:48.124398 10987 ts_tablet_manager.cc:1918] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:48.124486 10987 log.cc:1196] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/837e8a2179894512b24973eec6115428
I20260923 14:36:48.124747 10987 ts_tablet_manager.cc:1939] T 837e8a2179894512b24973eec6115428 P 71edd6118fe546aeabda4457aef8987f: Deleting consensus metadata
I20260923 14:36:48.125494 10986 ts_tablet_manager.cc:1918] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:48.125572 10986 log.cc:1196] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/837e8a2179894512b24973eec6115428
I20260923 14:36:48.125665 10438 catalog_manager.cc:4841] TS 71edd6118fe546aeabda4457aef8987f (127.2.218.2:46797): tablet 837e8a2179894512b24973eec6115428 (table test-table [id=1b8b0e59cea8469497c5d693fd2d020d]) successfully deleted
I20260923 14:36:48.125849 10986 ts_tablet_manager.cc:1939] T 837e8a2179894512b24973eec6115428 P 4bb5d88476fc45bcb840ef1862740ce5: Deleting consensus metadata
I20260923 14:36:48.126785 10437 catalog_manager.cc:4841] TS 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979): tablet 837e8a2179894512b24973eec6115428 (table test-table [id=1b8b0e59cea8469497c5d693fd2d020d]) successfully deleted
I20260923 14:36:48.127177 10985 ts_tablet_manager.cc:1918] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:48.127238 10985 log.cc:1196] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/837e8a2179894512b24973eec6115428
I20260923 14:36:48.127409 10985 ts_tablet_manager.cc:1939] T 837e8a2179894512b24973eec6115428 P 92697923160649df98e76ba50b1bdf1a: Deleting consensus metadata
I20260923 14:36:48.128286 10436 catalog_manager.cc:4841] TS 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657): tablet 837e8a2179894512b24973eec6115428 (table test-table [id=1b8b0e59cea8469497c5d693fd2d020d]) successfully deleted
I20260923 14:36:48.128674 10962 tablet_bootstrap.cc:492] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: No bootstrap required, opened a new log
I20260923 14:36:48.128733 10962 ts_tablet_manager.cc:1397] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260923 14:36:48.129070 10962 raft_consensus.cc:348] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.129172 10962 raft_consensus.cc:374] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.129192 10962 raft_consensus.cc:729] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71edd6118fe546aeabda4457aef8987f, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.129302 10962 consensus_queue.cc:260] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.129539 10962 ts_tablet_manager.cc:1428] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:48.140545 10972 raft_consensus.cc:482] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:48.140642 10972 raft_consensus.cc:504] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.141058 10972 leader_election.cc:288] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979), 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657)
I20260923 14:36:48.141367 10874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c595d965f2d46839d2d65cb69b543b8" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92697923160649df98e76ba50b1bdf1a" is_pre_election: true
I20260923 14:36:48.141369 10608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c595d965f2d46839d2d65cb69b543b8" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb5d88476fc45bcb840ef1862740ce5" is_pre_election: true
I20260923 14:36:48.141479 10608 raft_consensus.cc:2454] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 0.
I20260923 14:36:48.141501 10874 raft_consensus.cc:2454] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 0.
I20260923 14:36:48.141726 10681 leader_election.cc:302] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb5d88476fc45bcb840ef1862740ce5, 71edd6118fe546aeabda4457aef8987f; no voters:
I20260923 14:36:48.141896 10972 raft_consensus.cc:2781] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:48.141950 10972 raft_consensus.cc:482] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:48.141989 10972 raft_consensus.cc:3037] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.142812 10972 raft_consensus.cc:504] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.143162 10972 leader_election.cc:288] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 election: Requested vote from peers 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979), 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657)
I20260923 14:36:48.143388 10608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c595d965f2d46839d2d65cb69b543b8" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb5d88476fc45bcb840ef1862740ce5"
I20260923 14:36:48.143460 10874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c595d965f2d46839d2d65cb69b543b8" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92697923160649df98e76ba50b1bdf1a"
I20260923 14:36:48.143518 10608 raft_consensus.cc:3037] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.143553 10874 raft_consensus.cc:3037] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.144277 10608 raft_consensus.cc:2454] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 1.
I20260923 14:36:48.144419 10874 raft_consensus.cc:2454] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 1.
I20260923 14:36:48.144526 10681 leader_election.cc:302] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb5d88476fc45bcb840ef1862740ce5, 71edd6118fe546aeabda4457aef8987f; no voters:
I20260923 14:36:48.144693 10972 raft_consensus.cc:2781] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:48.144778 10972 raft_consensus.cc:686] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 1 LEADER]: Becoming Leader. State: Replica: 71edd6118fe546aeabda4457aef8987f, State: Running, Role: LEADER
I20260923 14:36:48.144881 10972 consensus_queue.cc:237] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.146224 10451 catalog_manager.cc:5495] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f reported cstate change: term changed from 0 to 1, leader changed from <none> to 71edd6118fe546aeabda4457aef8987f (127.2.218.2). New cstate: current_term: 1 leader_uuid: "71edd6118fe546aeabda4457aef8987f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } health_report { overall_health: UNKNOWN } } }
W20260923 14:36:48.161793 10795 tablet.cc:2369] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 14:36:48 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174207, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.62@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:48.187573 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.179566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38257 (local address 127.0.0.1:39884)
0923 14:36:48.179813 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.179839 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.179917 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.180141 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.180142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.180473 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:48.180553 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.180558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.181470 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.181472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.181947 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.181954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.182472 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.182475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.182543 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.183167 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:48.183178 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:48.184562 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:48.186269 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:48.186277 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:48.186281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:48.186475 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:48.186769 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:48.186771 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:48.186772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:48.186841 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:48.187200 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:48.187205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:48.187288 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.187366 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.187458 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260923 14:36:48.187857 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.179658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.2.218.62:38257)
0923 14:36:48.179775 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.179779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.179899 (+ 120us) server_negotiation.cc:407] Connection header received
0923 14:36:48.179984 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.179986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.180024 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.180099 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:48.180644 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.181357 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.182055 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.182344 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.182440 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.184697 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:48.184701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:48.184703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:48.184725 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:48.186154 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:48.186585 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:48.186588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:48.186590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:48.186684 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:48.186920 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:48.186923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:48.186924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:48.187098 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:48.187186 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.187432 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.187531 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:36:48.191125 10451 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:39884:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 14:36:48.191763 10451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:48.196097 10588 tablet_service.cc:1467] Processing CreateTablet for tablet 3bac00d265174ed78078bae2b4ea65b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae7b79e3a3f5484d9dd6ebcf0623da2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:36:48.196173 10728 tablet_service.cc:1467] Processing CreateTablet for tablet 3bac00d265174ed78078bae2b4ea65b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae7b79e3a3f5484d9dd6ebcf0623da2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:36:48.196260 10868 tablet_service.cc:1467] Processing CreateTablet for tablet 3bac00d265174ed78078bae2b4ea65b5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae7b79e3a3f5484d9dd6ebcf0623da2e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 14:36:48.196441 10588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bac00d265174ed78078bae2b4ea65b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.196475 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bac00d265174ed78078bae2b4ea65b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.196547 10868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bac00d265174ed78078bae2b4ea65b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.199033 10966 tablet_bootstrap.cc:492] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: Bootstrap starting.
I20260923 14:36:48.199056 10963 tablet_bootstrap.cc:492] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: Bootstrap starting.
I20260923 14:36:48.199921 10966 tablet_bootstrap.cc:654] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.199973 10963 tablet_bootstrap.cc:654] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.200498 10962 tablet_bootstrap.cc:492] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: Bootstrap starting.
I20260923 14:36:48.201125 10966 tablet_bootstrap.cc:492] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: No bootstrap required, opened a new log
I20260923 14:36:48.201221 10966 ts_tablet_manager.cc:1397] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:48.201365 10962 tablet_bootstrap.cc:654] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.201640 10966 raft_consensus.cc:348] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.201766 10966 raft_consensus.cc:374] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.201814 10966 raft_consensus.cc:729] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92697923160649df98e76ba50b1bdf1a, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.201944 10966 consensus_queue.cc:260] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.202240 10966 ts_tablet_manager.cc:1428] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:48.202428 10970 tablet_replica.cc:439] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } }
I20260923 14:36:48.202549 10970 tablet_replica.cc:442] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:48.202917 10963 tablet_bootstrap.cc:492] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: No bootstrap required, opened a new log
I20260923 14:36:48.202977 10963 ts_tablet_manager.cc:1397] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260923 14:36:48.203244 10962 tablet_bootstrap.cc:492] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: No bootstrap required, opened a new log
I20260923 14:36:48.203321 10962 ts_tablet_manager.cc:1397] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 14:36:48.203383 10963 raft_consensus.cc:348] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.203475 10963 raft_consensus.cc:374] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.203500 10963 raft_consensus.cc:729] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb5d88476fc45bcb840ef1862740ce5, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.203593 10963 consensus_queue.cc:260] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.203814 10963 ts_tablet_manager.cc:1428] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:48.203948 10962 raft_consensus.cc:348] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.204035 10962 raft_consensus.cc:374] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.204070 10962 raft_consensus.cc:729] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71edd6118fe546aeabda4457aef8987f, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.204017 10969 tablet_replica.cc:439] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } }
I20260923 14:36:48.204126 10969 tablet_replica.cc:442] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:48.204183 10962 consensus_queue.cc:260] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.204566 10962 ts_tablet_manager.cc:1428] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:48.205044 10972 tablet_replica.cc:439] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } }
I20260923 14:36:48.205152 10972 tablet_replica.cc:442] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:48.311473 10970 raft_consensus.cc:482] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:48.311590 10970 raft_consensus.cc:504] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.312603 10970 leader_election.cc:288] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979), 71edd6118fe546aeabda4457aef8987f (127.2.218.2:46797)
I20260923 14:36:48.316287 10827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.312836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:46797 (local address 127.2.218.3:38459)
0923 14:36:48.313021 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.313035 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.313127 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.313391 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.313398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.313414 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:48.313543 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.313552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.314316 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.314319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.315396 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.315405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.315964 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.315967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.316029 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.316046 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.316098 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.316144 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260923 14:36:48.316390 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.312924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:38459 (local address 127.2.218.2:46797)
0923 14:36:48.313035 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.313039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.313171 (+ 132us) server_negotiation.cc:407] Connection header received
0923 14:36:48.313222 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.313225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.313262 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.313362 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:48.313664 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.314215 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.315503 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.315867 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.315953 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.316009 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.316128 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.316222 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:36:48.316818 10748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bac00d265174ed78078bae2b4ea65b5" candidate_uuid: "92697923160649df98e76ba50b1bdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71edd6118fe546aeabda4457aef8987f" is_pre_election: true
I20260923 14:36:48.316906 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.312836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:37979 (local address 127.2.218.3:38989)
0923 14:36:48.313243 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.313253 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.313323 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.313572 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.313577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.313596 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:48.314033 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.314039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.314821 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.314824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.315769 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.315775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.316552 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.316556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.316657 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.316675 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.316744 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.316805 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":36,"thread_start_us":124,"threads_started":1}
I20260923 14:36:48.316977 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.312980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:38989 (local address 127.2.218.1:37979)
0923 14:36:48.313100 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.313105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.313339 (+ 234us) server_negotiation.cc:407] Connection header received
0923 14:36:48.313424 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.313427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.313481 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.313538 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:48.314167 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.314707 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.315871 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.316420 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.316523 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.316576 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.316786 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.316865 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:36:48.317037 10748 raft_consensus.cc:2454] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92697923160649df98e76ba50b1bdf1a in term 0.
I20260923 14:36:48.317457 10822 leader_election.cc:302] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71edd6118fe546aeabda4457aef8987f, 92697923160649df98e76ba50b1bdf1a; no voters:
I20260923 14:36:48.317471 10608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bac00d265174ed78078bae2b4ea65b5" candidate_uuid: "92697923160649df98e76ba50b1bdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb5d88476fc45bcb840ef1862740ce5" is_pre_election: true
I20260923 14:36:48.317605 10608 raft_consensus.cc:2454] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92697923160649df98e76ba50b1bdf1a in term 0.
I20260923 14:36:48.317716 10970 raft_consensus.cc:2781] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:48.317782 10970 raft_consensus.cc:482] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:48.317819 10970 raft_consensus.cc:3037] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.318686 10970 raft_consensus.cc:504] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.318957 10970 leader_election.cc:288] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [CANDIDATE]: Term 1 election: Requested vote from peers 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979), 71edd6118fe546aeabda4457aef8987f (127.2.218.2:46797)
I20260923 14:36:48.319281 10608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bac00d265174ed78078bae2b4ea65b5" candidate_uuid: "92697923160649df98e76ba50b1bdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb5d88476fc45bcb840ef1862740ce5"
I20260923 14:36:48.319280 10748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bac00d265174ed78078bae2b4ea65b5" candidate_uuid: "92697923160649df98e76ba50b1bdf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71edd6118fe546aeabda4457aef8987f"
I20260923 14:36:48.319377 10748 raft_consensus.cc:3037] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.319377 10608 raft_consensus.cc:3037] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.320310 10748 raft_consensus.cc:2454] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92697923160649df98e76ba50b1bdf1a in term 1.
I20260923 14:36:48.320310 10608 raft_consensus.cc:2454] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92697923160649df98e76ba50b1bdf1a in term 1.
I20260923 14:36:48.320560 10821 leader_election.cc:302] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb5d88476fc45bcb840ef1862740ce5, 92697923160649df98e76ba50b1bdf1a; no voters:
I20260923 14:36:48.320693 10970 raft_consensus.cc:2781] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:48.320834 10970 raft_consensus.cc:686] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [term 1 LEADER]: Becoming Leader. State: Replica: 92697923160649df98e76ba50b1bdf1a, State: Running, Role: LEADER
I20260923 14:36:48.321082 10970 consensus_queue.cc:237] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.322237 11002 tablet_replica.cc:439] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: TxnStatusTablet state changed. Reason: New leader 92697923160649df98e76ba50b1bdf1a. Latest consensus state: current_term: 1 leader_uuid: "92697923160649df98e76ba50b1bdf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } }
I20260923 14:36:48.322391 11002 tablet_replica.cc:442] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:36:48.322669 11004 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 14:36:48.322798 11004 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 14:36:48.323233 10451 catalog_manager.cc:5495] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 92697923160649df98e76ba50b1bdf1a (127.2.218.3). New cstate: current_term: 1 leader_uuid: "92697923160649df98e76ba50b1bdf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } health_report { overall_health: HEALTHY } } }
I20260923 14:36:48.357270 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.353509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:44657 (local address 127.0.0.1:42382)
0923 14:36:48.353638 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.353652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.353733 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.354051 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.354053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.354057 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:48.354189 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.354194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.355009 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.355011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.355626 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.355633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.356183 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.356186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.356249 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.356286 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.357002 (+ 716us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.357017 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.357085 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.357175 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 14:36:48.357273 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.353567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.2.218.3:44657)
0923 14:36:48.353664 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.353667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.353684 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:36:48.353862 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.353865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.353930 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.354050 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:48.354278 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.354913 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.355736 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.356066 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.356177 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.356420 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.356876 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.357013 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.357102 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.357150 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:36:48.359812 10608 raft_consensus.cc:1261] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Refusing update from remote peer 92697923160649df98e76ba50b1bdf1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:48.359839 10748 raft_consensus.cc:1261] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Refusing update from remote peer 92697923160649df98e76ba50b1bdf1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:48.360214 10970 consensus_queue.cc:1035] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:48.360340 11002 consensus_queue.cc:1035] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:48.361768 10969 tablet_replica.cc:439] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: TxnStatusTablet state changed. Reason: New leader 92697923160649df98e76ba50b1bdf1a. Latest consensus state: current_term: 1 leader_uuid: "92697923160649df98e76ba50b1bdf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } }
I20260923 14:36:48.361861 10969 tablet_replica.cc:442] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:48.361920 10972 tablet_replica.cc:439] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: TxnStatusTablet state changed. Reason: New leader 92697923160649df98e76ba50b1bdf1a. Latest consensus state: current_term: 1 leader_uuid: "92697923160649df98e76ba50b1bdf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } }
I20260923 14:36:48.362005 10972 tablet_replica.cc:442] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:48.363719 10970 tablet_replica.cc:439] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92697923160649df98e76ba50b1bdf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } }
I20260923 14:36:48.363813 10970 tablet_replica.cc:442] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:36:48.363929 10977 mvcc.cc:205] Tried to move back new op lower bound from 7332553557437804544 to 7332553557285285888. Current Snapshot: MvccSnapshot[applied={T|T < 7332553557437804544 or (T in {7332553557437804544})}]
I20260923 14:36:48.366171 11002 tablet_replica.cc:439] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92697923160649df98e76ba50b1bdf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } }
I20260923 14:36:48.366214 10975 mvcc.cc:205] Tried to move back new op lower bound from 7332553557437804544 to 7332553557285285888. Current Snapshot: MvccSnapshot[applied={T|T < 7332553557437804544}]
I20260923 14:36:48.366250 11002 tablet_replica.cc:442] T 3bac00d265174ed78078bae2b4ea65b5 P 92697923160649df98e76ba50b1bdf1a: This TxnStatusTablet replica's current role is: LEADER
I20260923 14:36:48.366595 10972 tablet_replica.cc:439] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92697923160649df98e76ba50b1bdf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } }
I20260923 14:36:48.366701 10972 tablet_replica.cc:442] T 3bac00d265174ed78078bae2b4ea65b5 P 71edd6118fe546aeabda4457aef8987f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:48.368566 10969 tablet_replica.cc:439] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92697923160649df98e76ba50b1bdf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } } }
I20260923 14:36:48.368661 10969 tablet_replica.cc:442] T 3bac00d265174ed78078bae2b4ea65b5 P 4bb5d88476fc45bcb840ef1862740ce5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 14:36:48.376607 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.373687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:55995 (local address 127.2.218.3:44657)
0923 14:36:48.373779 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.373783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.373993 (+ 210us) server_negotiation.cc:407] Connection header received
0923 14:36:48.374148 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.374151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.374189 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.374247 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:48.374475 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.374853 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.375858 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.376019 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.376074 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.376331 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.376379 (+ 48us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.376420 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.376512 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.376540 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 14:36:48.376698 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.373569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:44657 (local address 127.2.218.2:55995)
0923 14:36:48.373907 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.373920 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.374021 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.374278 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.374281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.374289 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:48.374391 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.374396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.374938 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.374941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.375756 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.375765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.376101 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.376104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.376184 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.376225 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.376444 (+ 219us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.376449 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.376509 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.376578 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":87,"threads_started":1}
I20260923 14:36:48.381978 10608 raft_consensus.cc:1261] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Refusing update from remote peer 71edd6118fe546aeabda4457aef8987f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:48.381995 10874 raft_consensus.cc:1261] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Refusing update from remote peer 71edd6118fe546aeabda4457aef8987f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:48.382323 10972 consensus_queue.cc:1035] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:48.382488 10967 consensus_queue.cc:1035] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:48.399194 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.396209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:36309 (local address 127.2.218.2:46797)
0923 14:36:48.396301 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.396305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.396317 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:48.396391 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.396393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.396428 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.396490 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:48.396712 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.397098 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.398164 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.398388 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.398497 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.398776 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.398847 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.398886 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.399058 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.399104 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 14:36:48.399367 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.395947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:46797 (local address 127.2.218.3:36309)
0923 14:36:48.396235 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.396247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.396322 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.396521 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.396524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.396529 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:48.396637 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.396642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.397192 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.397194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.398011 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.398020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.398504 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.398508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.398592 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.398632 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.398959 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.398964 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.399037 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.399100 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
I20260923 14:36:48.423301 10451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:48.423432 10451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:48.425127 10451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c: Sending DeleteTablet for 3 replicas of tablet 7c595d965f2d46839d2d65cb69b543b8
I20260923 14:36:48.425666 10588 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c595d965f2d46839d2d65cb69b543b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:47880
I20260923 14:36:48.425827 10986 tablet_replica.cc:330] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5: stopping tablet replica
I20260923 14:36:48.425841 10728 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c595d965f2d46839d2d65cb69b543b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:33234
I20260923 14:36:48.425935 10986 raft_consensus.cc:2229] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:48.425969 10987 tablet_replica.cc:330] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: stopping tablet replica
I20260923 14:36:48.426035 10986 raft_consensus.cc:2258] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.426055 10987 raft_consensus.cc:2229] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:48.426369 10987 raft_consensus.cc:2258] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.428748 10868 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c595d965f2d46839d2d65cb69b543b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:42366
I20260923 14:36:48.428910 10985 tablet_replica.cc:330] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a: stopping tablet replica
I20260923 14:36:48.429003 10985 raft_consensus.cc:2229] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:48.429090 10985 raft_consensus.cc:2258] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.429137 10986 ts_tablet_manager.cc:1905] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.429575 10985 ts_tablet_manager.cc:1905] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.430514 10987 ts_tablet_manager.cc:1905] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.431744 10986 ts_tablet_manager.cc:1918] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:48.431811 10986 log.cc:1196] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/7c595d965f2d46839d2d65cb69b543b8
I20260923 14:36:48.431874 10985 ts_tablet_manager.cc:1918] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:48.431937 10985 log.cc:1196] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/7c595d965f2d46839d2d65cb69b543b8
I20260923 14:36:48.432049 10986 ts_tablet_manager.cc:1939] T 7c595d965f2d46839d2d65cb69b543b8 P 4bb5d88476fc45bcb840ef1862740ce5: Deleting consensus metadata
I20260923 14:36:48.432188 10985 ts_tablet_manager.cc:1939] T 7c595d965f2d46839d2d65cb69b543b8 P 92697923160649df98e76ba50b1bdf1a: Deleting consensus metadata
I20260923 14:36:48.432884 10437 catalog_manager.cc:4841] TS 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979): tablet 7c595d965f2d46839d2d65cb69b543b8 (table test-table [id=08e4d1517c2e44b3a1b8e05263c88f22]) successfully deleted
I20260923 14:36:48.432953 10987 ts_tablet_manager.cc:1918] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:48.432996 10987 log.cc:1196] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/7c595d965f2d46839d2d65cb69b543b8
I20260923 14:36:48.433285 10987 ts_tablet_manager.cc:1939] T 7c595d965f2d46839d2d65cb69b543b8 P 71edd6118fe546aeabda4457aef8987f: Deleting consensus metadata
I20260923 14:36:48.433264 10436 catalog_manager.cc:4841] TS 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657): tablet 7c595d965f2d46839d2d65cb69b543b8 (table test-table [id=08e4d1517c2e44b3a1b8e05263c88f22]) successfully deleted
I20260923 14:36:48.433871 10438 catalog_manager.cc:4841] TS 71edd6118fe546aeabda4457aef8987f (127.2.218.2:46797): tablet 7c595d965f2d46839d2d65cb69b543b8 (table test-table [id=08e4d1517c2e44b3a1b8e05263c88f22]) successfully deleted
Sep 23 14:36:48 dist-test-slave-zl47 krb5kdc[10406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 14:36:48 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:48 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 14:36:48.453457 10500 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:48.447349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.2.218.62:38257)
0923 14:36:48.447473 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.447477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.447659 (+ 182us) server_negotiation.cc:407] Connection header received
0923 14:36:48.447788 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.447790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.447824 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.447880 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:48.448668 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.449168 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.450279 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.450478 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.450545 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.453344 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39898: BlockingRecv error: recv got EOF from 127.0.0.1:39898 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 23 14:36:48 dist-test-slave-zl47 krb5kdc[10406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM
I20260923 14:36:48.462138 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.454793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.2.218.62:38257)
0923 14:36:48.454873 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.454876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.454987 (+ 111us) server_negotiation.cc:407] Connection header received
0923 14:36:48.455056 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.455059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.455092 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.455153 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:48.455593 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.456226 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.456838 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.457108 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.457194 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.459241 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:48.459245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:48.459250 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:48.459278 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:48.460609 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:48.461020 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:48.461023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:48.461025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:48.461114 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:48.461386 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:48.461389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:48.461390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:48.461568 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:48.461678 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.461888 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.461981 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 14:36:48.464712 10451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:36:48.465128 10451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:48.470525 10728 tablet_service.cc:1467] Processing CreateTablet for tablet 33564e76096649b080a7e18537c4e5fa (DEFAULT_TABLE table=test-table [id=61d1ac87c22d4611b702a6ce8da11eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:48.470705 10868 tablet_service.cc:1467] Processing CreateTablet for tablet 33564e76096649b080a7e18537c4e5fa (DEFAULT_TABLE table=test-table [id=61d1ac87c22d4611b702a6ce8da11eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:48.470764 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33564e76096649b080a7e18537c4e5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.470916 10868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33564e76096649b080a7e18537c4e5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.472882 10962 tablet_bootstrap.cc:492] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f: Bootstrap starting.
I20260923 14:36:48.473680 10962 tablet_bootstrap.cc:654] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.474203 10966 tablet_bootstrap.cc:492] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a: Bootstrap starting.
I20260923 14:36:48.474867 10966 tablet_bootstrap.cc:654] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.475518 10588 tablet_service.cc:1467] Processing CreateTablet for tablet 33564e76096649b080a7e18537c4e5fa (DEFAULT_TABLE table=test-table [id=61d1ac87c22d4611b702a6ce8da11eb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:48.475739 10588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33564e76096649b080a7e18537c4e5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.477730 10963 tablet_bootstrap.cc:492] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5: Bootstrap starting.
I20260923 14:36:48.478446 10963 tablet_bootstrap.cc:654] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.478660 10966 tablet_bootstrap.cc:492] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a: No bootstrap required, opened a new log
I20260923 14:36:48.478725 10966 ts_tablet_manager.cc:1397] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260923 14:36:48.479099 10966 raft_consensus.cc:348] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.479219 10966 raft_consensus.cc:374] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.479249 10966 raft_consensus.cc:729] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92697923160649df98e76ba50b1bdf1a, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.479338 10962 tablet_bootstrap.cc:492] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f: No bootstrap required, opened a new log
I20260923 14:36:48.479355 10966 consensus_queue.cc:260] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.479413 10962 ts_tablet_manager.cc:1397] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260923 14:36:48.479575 10966 ts_tablet_manager.cc:1428] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:48.479764 10962 raft_consensus.cc:348] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.479867 10962 raft_consensus.cc:374] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.479897 10962 raft_consensus.cc:729] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71edd6118fe546aeabda4457aef8987f, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.480002 10962 consensus_queue.cc:260] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.480199 10963 tablet_bootstrap.cc:492] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5: No bootstrap required, opened a new log
I20260923 14:36:48.480265 10962 ts_tablet_manager.cc:1428] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:48.480280 10963 ts_tablet_manager.cc:1397] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 14:36:48.480829 10963 raft_consensus.cc:348] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.480973 10963 raft_consensus.cc:374] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.481003 10963 raft_consensus.cc:729] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb5d88476fc45bcb840ef1862740ce5, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.481113 10963 consensus_queue.cc:260] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.481351 10963 ts_tablet_manager.cc:1428] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:48.514361 10972 raft_consensus.cc:482] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:48.514472 10972 raft_consensus.cc:504] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.514798 10972 leader_election.cc:288] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979), 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657)
I20260923 14:36:48.515134 10874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33564e76096649b080a7e18537c4e5fa" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92697923160649df98e76ba50b1bdf1a" is_pre_election: true
I20260923 14:36:48.515143 10608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33564e76096649b080a7e18537c4e5fa" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb5d88476fc45bcb840ef1862740ce5" is_pre_election: true
I20260923 14:36:48.515241 10874 raft_consensus.cc:2454] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 0.
I20260923 14:36:48.515267 10608 raft_consensus.cc:2454] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 0.
I20260923 14:36:48.515525 10680 leader_election.cc:302] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71edd6118fe546aeabda4457aef8987f, 92697923160649df98e76ba50b1bdf1a; no voters:
I20260923 14:36:48.515684 10972 raft_consensus.cc:2781] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:48.515738 10972 raft_consensus.cc:482] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:48.515779 10972 raft_consensus.cc:3037] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.516552 10972 raft_consensus.cc:504] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.516805 10972 leader_election.cc:288] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 election: Requested vote from peers 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979), 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657)
I20260923 14:36:48.517019 10608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33564e76096649b080a7e18537c4e5fa" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb5d88476fc45bcb840ef1862740ce5"
I20260923 14:36:48.517112 10608 raft_consensus.cc:3037] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.517105 10874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33564e76096649b080a7e18537c4e5fa" candidate_uuid: "71edd6118fe546aeabda4457aef8987f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92697923160649df98e76ba50b1bdf1a"
I20260923 14:36:48.517261 10874 raft_consensus.cc:3037] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.518229 10608 raft_consensus.cc:2454] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 1.
I20260923 14:36:48.518328 10874 raft_consensus.cc:2454] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71edd6118fe546aeabda4457aef8987f in term 1.
I20260923 14:36:48.518513 10681 leader_election.cc:302] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb5d88476fc45bcb840ef1862740ce5, 71edd6118fe546aeabda4457aef8987f; no voters:
I20260923 14:36:48.518688 10972 raft_consensus.cc:2781] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:48.518832 10972 raft_consensus.cc:686] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 1 LEADER]: Becoming Leader. State: Replica: 71edd6118fe546aeabda4457aef8987f, State: Running, Role: LEADER
I20260923 14:36:48.518985 10972 consensus_queue.cc:237] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.520288 10451 catalog_manager.cc:5495] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f reported cstate change: term changed from 0 to 1, leader changed from <none> to 71edd6118fe546aeabda4457aef8987f (127.2.218.2). New cstate: current_term: 1 leader_uuid: "71edd6118fe546aeabda4457aef8987f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } health_report { overall_health: UNKNOWN } } }
I20260923 14:36:48.533867 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.531017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.2.218.2:46797)
0923 14:36:48.531129 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.531132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.531158 (+ 26us) server_negotiation.cc:407] Connection header received
0923 14:36:48.531229 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.531232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.531282 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.531346 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:48.531557 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.531984 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.532885 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.533094 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.533205 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.533493 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.533551 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.533622 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.533741 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.533785 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:36:48.535516 10874 raft_consensus.cc:1261] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Refusing update from remote peer 71edd6118fe546aeabda4457aef8987f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:48.535629 10608 raft_consensus.cc:1261] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Refusing update from remote peer 71edd6118fe546aeabda4457aef8987f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:48.536036 10967 consensus_queue.cc:1035] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [LEADER]: Connected to new peer: Peer: permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:48.536161 10972 consensus_queue.cc:1035] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:48.541282 10451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:48.541391 10451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:48.543126 10451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c: Sending DeleteTablet for 3 replicas of tablet 33564e76096649b080a7e18537c4e5fa
I20260923 14:36:48.543546 10588 tablet_service.cc:1514] Processing DeleteTablet for tablet 33564e76096649b080a7e18537c4e5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:47880
I20260923 14:36:48.543694 10728 tablet_service.cc:1514] Processing DeleteTablet for tablet 33564e76096649b080a7e18537c4e5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:33234
I20260923 14:36:48.543705 10868 tablet_service.cc:1514] Processing DeleteTablet for tablet 33564e76096649b080a7e18537c4e5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:42366
I20260923 14:36:48.543731 10986 tablet_replica.cc:330] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5: stopping tablet replica
I20260923 14:36:48.543833 10986 raft_consensus.cc:2229] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:48.543853 10987 tablet_replica.cc:330] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f: stopping tablet replica
I20260923 14:36:48.543941 10987 raft_consensus.cc:2229] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:48.543942 10986 raft_consensus.cc:2258] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.544072 10987 raft_consensus.cc:2258] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.544253 10985 tablet_replica.cc:330] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a: stopping tablet replica
I20260923 14:36:48.544463 10985 raft_consensus.cc:2229] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:48.544582 10985 raft_consensus.cc:2258] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.544901 10986 ts_tablet_manager.cc:1905] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.544847 10451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 14:36:48.545127 10987 ts_tablet_manager.cc:1905] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 14:36:48.545208 10451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:36:48.545230 10985 ts_tablet_manager.cc:1905] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.547029 10986 ts_tablet_manager.cc:1918] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:48.547114 10986 log.cc:1196] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/33564e76096649b080a7e18537c4e5fa
I20260923 14:36:48.547122 10985 ts_tablet_manager.cc:1918] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:48.547122 10987 ts_tablet_manager.cc:1918] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:36:48.547189 10985 log.cc:1196] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/33564e76096649b080a7e18537c4e5fa
I20260923 14:36:48.547189 10987 log.cc:1196] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/33564e76096649b080a7e18537c4e5fa
I20260923 14:36:48.547351 10986 ts_tablet_manager.cc:1939] T 33564e76096649b080a7e18537c4e5fa P 4bb5d88476fc45bcb840ef1862740ce5: Deleting consensus metadata
I20260923 14:36:48.547417 10985 ts_tablet_manager.cc:1939] T 33564e76096649b080a7e18537c4e5fa P 92697923160649df98e76ba50b1bdf1a: Deleting consensus metadata
I20260923 14:36:48.547430 10987 ts_tablet_manager.cc:1939] T 33564e76096649b080a7e18537c4e5fa P 71edd6118fe546aeabda4457aef8987f: Deleting consensus metadata
I20260923 14:36:48.547955 10437 catalog_manager.cc:4841] TS 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979): tablet 33564e76096649b080a7e18537c4e5fa (table test-table [id=61d1ac87c22d4611b702a6ce8da11eb0]) successfully deleted
I20260923 14:36:48.547983 10436 catalog_manager.cc:4841] TS 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657): tablet 33564e76096649b080a7e18537c4e5fa (table test-table [id=61d1ac87c22d4611b702a6ce8da11eb0]) successfully deleted
I20260923 14:36:48.548115 10438 catalog_manager.cc:4841] TS 71edd6118fe546aeabda4457aef8987f (127.2.218.2:46797): tablet 33564e76096649b080a7e18537c4e5fa (table test-table [id=61d1ac87c22d4611b702a6ce8da11eb0]) successfully deleted
I20260923 14:36:48.550644 10588 tablet_service.cc:1467] Processing CreateTablet for tablet 59f6db305d2844c9a3e254029be2d195 (DEFAULT_TABLE table=test-table [id=16a08dcb17b74c3bb7b076fb3d850143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:48.550701 10728 tablet_service.cc:1467] Processing CreateTablet for tablet 59f6db305d2844c9a3e254029be2d195 (DEFAULT_TABLE table=test-table [id=16a08dcb17b74c3bb7b076fb3d850143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:48.550880 10588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f6db305d2844c9a3e254029be2d195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.550915 10728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f6db305d2844c9a3e254029be2d195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.552358 10962 tablet_bootstrap.cc:492] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f: Bootstrap starting.
I20260923 14:36:48.552423 10963 tablet_bootstrap.cc:492] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5: Bootstrap starting.
I20260923 14:36:48.552951 10962 tablet_bootstrap.cc:654] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.553047 10963 tablet_bootstrap.cc:654] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.553431 10868 tablet_service.cc:1467] Processing CreateTablet for tablet 59f6db305d2844c9a3e254029be2d195 (DEFAULT_TABLE table=test-table [id=16a08dcb17b74c3bb7b076fb3d850143]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:36:48.553625 10868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59f6db305d2844c9a3e254029be2d195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:48.554055 10963 tablet_bootstrap.cc:492] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5: No bootstrap required, opened a new log
I20260923 14:36:48.554173 10963 ts_tablet_manager.cc:1397] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:48.554562 10963 raft_consensus.cc:348] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.554652 10963 raft_consensus.cc:374] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.554701 10963 raft_consensus.cc:729] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb5d88476fc45bcb840ef1862740ce5, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.554816 10963 consensus_queue.cc:260] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.555052 10963 ts_tablet_manager.cc:1428] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 14:36:48.555429 10966 tablet_bootstrap.cc:492] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a: Bootstrap starting.
I20260923 14:36:48.555732 10962 tablet_bootstrap.cc:492] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f: No bootstrap required, opened a new log
I20260923 14:36:48.555811 10962 ts_tablet_manager.cc:1397] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 14:36:48.556113 10966 tablet_bootstrap.cc:654] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:48.556155 10962 raft_consensus.cc:348] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.556242 10962 raft_consensus.cc:374] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.556274 10962 raft_consensus.cc:729] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71edd6118fe546aeabda4457aef8987f, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.556381 10962 consensus_queue.cc:260] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.556609 10962 ts_tablet_manager.cc:1428] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:48.557416 10966 tablet_bootstrap.cc:492] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a: No bootstrap required, opened a new log
I20260923 14:36:48.557511 10966 ts_tablet_manager.cc:1397] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:48.557828 10966 raft_consensus.cc:348] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.557905 10966 raft_consensus.cc:374] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:48.557945 10966 raft_consensus.cc:729] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92697923160649df98e76ba50b1bdf1a, State: Initialized, Role: FOLLOWER
I20260923 14:36:48.558051 10966 consensus_queue.cc:260] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.558293 10966 ts_tablet_manager.cc:1428] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 14:36:48.693681 10969 raft_consensus.cc:482] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:36:48.693797 10969 raft_consensus.cc:504] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.694700 10969 leader_election.cc:288] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71edd6118fe546aeabda4457aef8987f (127.2.218.2:46797), 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657)
I20260923 14:36:48.699558 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.695011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:58433 (local address 127.2.218.3:44657)
0923 14:36:48.695111 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.695115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.695493 (+ 378us) server_negotiation.cc:407] Connection header received
0923 14:36:48.695591 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.695593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.695639 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.695708 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:48.696089 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.696773 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.698369 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.698895 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.698953 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.699006 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.699371 (+ 365us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.699443 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:36:48.699556 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.695003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:57781 (local address 127.2.218.2:46797)
0923 14:36:48.695110 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.695115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.695408 (+ 293us) server_negotiation.cc:407] Connection header received
0923 14:36:48.695551 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.695556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.695621 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.695699 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:36:48.696089 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.696779 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.698369 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.698879 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.698952 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.699003 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.699371 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.699443 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:36:48.699605 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.694946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:44657 (local address 127.2.218.1:58433)
0923 14:36:48.695383 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.695394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.695488 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.695733 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.695735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.695756 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:48.695970 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.695977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.696902 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.696905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.698239 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.698249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.699103 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.699106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.699217 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.699264 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.699335 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.699435 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":263,"spinlock_wait_cycles":1792,"thread_start_us":80,"threads_started":1}
I20260923 14:36:48.699755 11041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.694946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:46797 (local address 127.2.218.1:57781)
0923 14:36:48.695302 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.695316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.695427 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.695733 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.695735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.695764 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:36:48.695991 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.695996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.696902 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.696905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.698239 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.698249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.699095 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.699099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.699243 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.699279 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.699335 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.699435 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":231,"thread_start_us":127,"threads_started":1}
I20260923 14:36:48.700058 10874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f6db305d2844c9a3e254029be2d195" candidate_uuid: "4bb5d88476fc45bcb840ef1862740ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92697923160649df98e76ba50b1bdf1a" is_pre_election: true
I20260923 14:36:48.700189 10874 raft_consensus.cc:2454] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bb5d88476fc45bcb840ef1862740ce5 in term 0.
I20260923 14:36:48.700356 10748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f6db305d2844c9a3e254029be2d195" candidate_uuid: "4bb5d88476fc45bcb840ef1862740ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71edd6118fe546aeabda4457aef8987f" is_pre_election: true
I20260923 14:36:48.700510 10748 raft_consensus.cc:2454] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bb5d88476fc45bcb840ef1862740ce5 in term 0.
I20260923 14:36:48.700517 10540 leader_election.cc:302] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb5d88476fc45bcb840ef1862740ce5, 92697923160649df98e76ba50b1bdf1a; no voters:
I20260923 14:36:48.700789 10969 raft_consensus.cc:2781] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:36:48.700840 10969 raft_consensus.cc:482] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:36:48.700896 10969 raft_consensus.cc:3037] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.701727 10969 raft_consensus.cc:504] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.702078 10969 leader_election.cc:288] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [CANDIDATE]: Term 1 election: Requested vote from peers 71edd6118fe546aeabda4457aef8987f (127.2.218.2:46797), 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657)
I20260923 14:36:48.702406 10874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f6db305d2844c9a3e254029be2d195" candidate_uuid: "4bb5d88476fc45bcb840ef1862740ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92697923160649df98e76ba50b1bdf1a"
I20260923 14:36:48.702522 10874 raft_consensus.cc:3037] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.702553 10748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59f6db305d2844c9a3e254029be2d195" candidate_uuid: "4bb5d88476fc45bcb840ef1862740ce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71edd6118fe546aeabda4457aef8987f"
I20260923 14:36:48.702654 10748 raft_consensus.cc:3037] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:48.703392 10874 raft_consensus.cc:2454] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb5d88476fc45bcb840ef1862740ce5 in term 1.
I20260923 14:36:48.703593 10748 raft_consensus.cc:2454] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb5d88476fc45bcb840ef1862740ce5 in term 1.
I20260923 14:36:48.703636 10540 leader_election.cc:302] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb5d88476fc45bcb840ef1862740ce5, 92697923160649df98e76ba50b1bdf1a; no voters:
I20260923 14:36:48.703768 10969 raft_consensus.cc:2781] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:48.703924 10969 raft_consensus.cc:686] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 LEADER]: Becoming Leader. State: Replica: 4bb5d88476fc45bcb840ef1862740ce5, State: Running, Role: LEADER
I20260923 14:36:48.704101 10969 consensus_queue.cc:237] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } }
I20260923 14:36:48.706014 10449 catalog_manager.cc:5495] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1). New cstate: current_term: 1 leader_uuid: "4bb5d88476fc45bcb840ef1862740ce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bb5d88476fc45bcb840ef1862740ce5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 37979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 } health_report { overall_health: UNKNOWN } } }
I20260923 14:36:48.745811 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.742712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.2.218.1:37979)
0923 14:36:48.742831 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.742835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.742846 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:36:48.743012 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.743015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.743068 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.743164 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:48.743373 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.743814 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.744548 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.744760 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.744852 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.745086 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.745477 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.745586 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.745648 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.745691 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:36:48.754238 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.751495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:44657 (local address 127.2.218.1:47095)
0923 14:36:48.751794 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.751804 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.751875 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.752068 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.752069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.752074 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:48.752151 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.752155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.752738 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.752740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.753233 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.753239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.753604 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.753606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.753672 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.753704 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.753963 (+ 259us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.753968 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.754015 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.754066 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":96,"threads_started":1}
I20260923 14:36:48.754233 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.751602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:47095 (local address 127.2.218.3:44657)
0923 14:36:48.751703 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.751706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.751860 (+ 154us) server_negotiation.cc:407] Connection header received
0923 14:36:48.751945 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.751947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.751988 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.752050 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:48.752252 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.752659 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.753323 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.753514 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.753592 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.753815 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.753891 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.753946 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.754040 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.754146 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:36:48.759054 10874 raft_consensus.cc:1261] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Refusing update from remote peer 4bb5d88476fc45bcb840ef1862740ce5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:48.759136 10748 raft_consensus.cc:1261] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Refusing update from remote peer 4bb5d88476fc45bcb840ef1862740ce5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:36:48.759531 11043 consensus_queue.cc:1035] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92697923160649df98e76ba50b1bdf1a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 44657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:48.759717 10969 consensus_queue.cc:1035] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71edd6118fe546aeabda4457aef8987f" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 46797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:36:48.775372 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.772540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:45997 (local address 127.2.218.1:37979)
0923 14:36:48.772659 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:48.772663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:48.772675 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:48.772706 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:48.772708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:48.772746 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:48.772809 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:36:48.773018 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.773600 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.774405 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.774593 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.774668 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.774953 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.775031 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.775097 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:48.775225 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 14:36:48.775276 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:36:48.775411 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:48.772389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:37979 (local address 127.2.218.3:45997)
0923 14:36:48.772516 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:48.772527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:48.772603 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:48.772826 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:48.772829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:48.772834 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 14:36:48.772912 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.772916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.773709 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.773711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.774298 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:48.774307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:48.774702 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:48.774706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:48.774798 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:48.774832 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 14:36:48.775131 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 14:36:48.775137 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 14:36:48.775191 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 14:36:48.775271 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 14:36:48.791761 10449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:48.791898 10449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:36:48.793659 10449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4947cd6bebd4fff91ccc97a7eadf92c: Sending DeleteTablet for 3 replicas of tablet 59f6db305d2844c9a3e254029be2d195
I20260923 14:36:48.794131 10728 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f6db305d2844c9a3e254029be2d195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:33234
I20260923 14:36:48.794100 10588 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f6db305d2844c9a3e254029be2d195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:47880
I20260923 14:36:48.794275 10986 tablet_replica.cc:330] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5: stopping tablet replica
I20260923 14:36:48.794275 10987 tablet_replica.cc:330] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f: stopping tablet replica
I20260923 14:36:48.794358 10987 raft_consensus.cc:2229] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:48.794358 10986 raft_consensus.cc:2229] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:36:48.794442 10987 raft_consensus.cc:2258] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.794581 10986 raft_consensus.cc:2258] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.795156 10986 ts_tablet_manager.cc:1905] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.795342 10987 ts_tablet_manager.cc:1905] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.797292 10986 ts_tablet_manager.cc:1918] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:48.797312 10987 ts_tablet_manager.cc:1918] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:48.797362 10987 log.cc:1196] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/59f6db305d2844c9a3e254029be2d195
I20260923 14:36:48.797362 10986 log.cc:1196] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/59f6db305d2844c9a3e254029be2d195
I20260923 14:36:48.797504 10868 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f6db305d2844c9a3e254029be2d195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:36:48 UTC) from {username='oryx', principal='oryx/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:42366
I20260923 14:36:48.797581 10986 ts_tablet_manager.cc:1939] T 59f6db305d2844c9a3e254029be2d195 P 4bb5d88476fc45bcb840ef1862740ce5: Deleting consensus metadata
I20260923 14:36:48.797581 10987 ts_tablet_manager.cc:1939] T 59f6db305d2844c9a3e254029be2d195 P 71edd6118fe546aeabda4457aef8987f: Deleting consensus metadata
I20260923 14:36:48.797664 10985 tablet_replica.cc:330] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a: stopping tablet replica
I20260923 14:36:48.797757 10985 raft_consensus.cc:2229] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:36:48.797840 10985 raft_consensus.cc:2258] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:36:48.798076 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 10523
I20260923 14:36:48.798085 10438 catalog_manager.cc:4841] TS 71edd6118fe546aeabda4457aef8987f (127.2.218.2:46797): tablet 59f6db305d2844c9a3e254029be2d195 (table test-table [id=16a08dcb17b74c3bb7b076fb3d850143]) successfully deleted
I20260923 14:36:48.799602 10437 catalog_manager.cc:4841] TS 4bb5d88476fc45bcb840ef1862740ce5 (127.2.218.1:37979): tablet 59f6db305d2844c9a3e254029be2d195 (table test-table [id=16a08dcb17b74c3bb7b076fb3d850143]) successfully deleted
I20260923 14:36:48.800352 10985 ts_tablet_manager.cc:1905] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:36:48.802497 10985 ts_tablet_manager.cc:1918] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 14:36:48.802551 10985 log.cc:1196] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/59f6db305d2844c9a3e254029be2d195
I20260923 14:36:48.802709 10985 ts_tablet_manager.cc:1939] T 59f6db305d2844c9a3e254029be2d195 P 92697923160649df98e76ba50b1bdf1a: Deleting consensus metadata
I20260923 14:36:48.803272 10436 catalog_manager.cc:4841] TS 92697923160649df98e76ba50b1bdf1a (127.2.218.3:44657): tablet 59f6db305d2844c9a3e254029be2d195 (table test-table [id=16a08dcb17b74c3bb7b076fb3d850143]) successfully deleted
I20260923 14:36:48.804701 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 10663
I20260923 14:36:48.810626 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 10803
I20260923 14:36:48.816195 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 10419
2026-09-23T14:36:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2737 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:48 dist-test-slave-zl47 krb5kdc[11054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:48 dist-test-slave-zl47 krb5kdc[11054](info): set up 2 sockets
Sep 23 14:36:48 dist-test-slave-zl47 krb5kdc[11054](info): commencing operation
krb5kdc: starting...
W20260923 14:36:49.711043 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.874s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:49 dist-test-slave-zl47 krb5kdc[11054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:49.810719 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:39707
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:39707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:49.902218 11067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:49.902450 11067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:49.902513 11067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:49.906301 11067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:49.906399 11067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:49.906437 11067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:49.906490 11067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:49.906544 11067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:49.911124 11067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:39707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:39707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:49.911906 11067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:49.912708 11067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:49.919749 11073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:49.919775 11072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:49.919886 11067 server_base.cc:1034] running on GCE node
W20260923 14:36:49.919785 11075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:49.920485 11067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:49.921432 11067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:49.922605 11067 hybrid_clock.cc:648] HybridClock initialized: now 1790174209922587 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T14:36:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1110 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:36:49 dist-test-slave-zl47 krb5kdc[11080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:36:49 dist-test-slave-zl47 krb5kdc[11080](info): set up 2 sockets
Sep 23 14:36:49 dist-test-slave-zl47 krb5kdc[11080](info): commencing operation
krb5kdc: starting...
W20260923 14:36:50.838829 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.891s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:36:50 dist-test-slave-zl47 krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:36:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:36:50.940261 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:38085
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:38085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:51.032944 11093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:51.033130 11093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:51.033161 11093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:51.036644 11093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:51.036689 11093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:51.036705 11093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:51.036720 11093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:51.036731 11093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:51.040920 11093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:38085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:38085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:51.041677 11093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:51.042466 11093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:51.049708 11101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:51.049733 11099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:51.049717 11098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:51.049767 11093 server_base.cc:1034] running on GCE node
I20260923 14:36:51.050396 11093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:51.051227 11093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:51.052448 11093 hybrid_clock.cc:648] HybridClock initialized: now 1790174211052434 us; error 30 us; skew 500 ppm
Sep 23 14:36:51 dist-test-slave-zl47 krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:51.057735 11093 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:51.058542 11093 webserver.cc:460] Webserver started at http://127.2.218.62:37139/ using document root <none> and password file <none>
I20260923 14:36:51.059000 11093 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:51.059079 11093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:51.059276 11093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:51.060927 11093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "e822827d55f1435baae081a5d5766e09"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "8f4f6e5c8edb51764a1c2f3e2965e6a8"
server_key_iv: "7d1ea2381326c94846fb9453fc816acd"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.061403 11093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "e822827d55f1435baae081a5d5766e09"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "8f4f6e5c8edb51764a1c2f3e2965e6a8"
server_key_iv: "7d1ea2381326c94846fb9453fc816acd"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.064429 11093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260923 14:36:51.066413 11108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.067432 11093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:51.067554 11093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "e822827d55f1435baae081a5d5766e09"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "8f4f6e5c8edb51764a1c2f3e2965e6a8"
server_key_iv: "7d1ea2381326c94846fb9453fc816acd"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.067646 11093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:51.090564 11093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:51.092780 11093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:51.092947 11093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:51.099463 11093 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:38085
I20260923 14:36:51.099483 11160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:38085 every 8 connection(s)
I20260923 14:36:51.100216 11093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:51.102470 11161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:36:51.104794 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 11093
I20260923 14:36:51.104926 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:36:51.105257 2920 external_mini_cluster.cc:1428] Setting key a5654476a4f17b5c60360514034fcc82
I20260923 14:36:51.107669 11161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09: Bootstrap starting.
I20260923 14:36:51.109537 11161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09: Neither blocks nor log segments found. Creating new log.
I20260923 14:36:51.110266 11161 log.cc:824] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:51.111927 11161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09: No bootstrap required, opened a new log
Sep 23 14:36:51 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:51.115629 11161 raft_consensus.cc:348] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } }
I20260923 14:36:51.115844 11161 raft_consensus.cc:374] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:36:51.115875 11161 raft_consensus.cc:729] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e822827d55f1435baae081a5d5766e09, State: Initialized, Role: FOLLOWER
I20260923 14:36:51.116250 11161 consensus_queue.cc:260] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } }
I20260923 14:36:51.116403 11161 raft_consensus.cc:388] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:51.116465 11161 raft_consensus.cc:482] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:51.116567 11161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:36:51.117303 11161 raft_consensus.cc:504] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } }
I20260923 14:36:51.117594 11161 leader_election.cc:302] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e822827d55f1435baae081a5d5766e09; no voters:
I20260923 14:36:51.117883 11161 leader_election.cc:288] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:36:51.117937 11166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:36:51.118134 11166 raft_consensus.cc:686] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 1 LEADER]: Becoming Leader. State: Replica: e822827d55f1435baae081a5d5766e09, State: Running, Role: LEADER
I20260923 14:36:51.118439 11166 consensus_queue.cc:237] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } }
I20260923 14:36:51.119016 11161 sys_catalog.cc:564] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:51.119490 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:51.106630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.2.218.62:38085)
0923 14:36:51.107016 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:51.107030 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:51.107062 (+ 32us) server_negotiation.cc:407] Connection header received
0923 14:36:51.107650 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:51.107665 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:51.107998 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:51.108317 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:51.109134 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.110207 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.111310 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.111752 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.111922 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:51.114507 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:51.114511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:51.114522 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:51.114546 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:51.116258 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:51.116823 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:51.116829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:51.116835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:51.116955 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:51.117261 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:51.117266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:51.117267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:51.117687 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:51.117908 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:51.118216 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 14:36:51.118626 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260923 14:36:51.120106 11167 sys_catalog.cc:455] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e822827d55f1435baae081a5d5766e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } } }
I20260923 14:36:51.120324 11167 sys_catalog.cc:458] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:51.119556 11166 sys_catalog.cc:455] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e822827d55f1435baae081a5d5766e09. Latest consensus state: current_term: 1 leader_uuid: "e822827d55f1435baae081a5d5766e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } } }
I20260923 14:36:51.120637 11166 sys_catalog.cc:458] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:51.122005 11175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:51.124397 11175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:51.129521 11175 catalog_manager.cc:1347] Generated new cluster ID: a965348fa7714a068bbcbcc463a8abd0
I20260923 14:36:51.129583 11175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:51.141446 11175 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:36:51.142262 11175 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:51.148078 11175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09: Generated new TSK 0
I20260923 14:36:51.148602 11175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:36:51.176033 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38085
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:51.266932 11187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:51.267115 11187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:51.267156 11187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:51.270548 11187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:51.270586 11187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:51.270632 11187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:51.274897 11187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:51.275576 11187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:51.276299 11187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:51.283951 11192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:51.284008 11187 server_base.cc:1034] running on GCE node
W20260923 14:36:51.284065 11193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:51.283963 11195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:51.284451 11187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:51.284969 11187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:51.286163 11187 hybrid_clock.cc:648] HybridClock initialized: now 1790174211286145 us; error 26 us; skew 500 ppm
Sep 23 14:36:51 dist-test-slave-zl47 krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:51.288470 11187 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:36:51.289093 11187 webserver.cc:460] Webserver started at http://127.2.218.1:37777/ using document root <none> and password file <none>
I20260923 14:36:51.289515 11187 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:51.289564 11187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:51.289695 11187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:51.291368 11187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "e8b82b91019a4f43b9b586ee2f58332f"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "37ca7ec389e1619f0f7cae91b20f81f7"
server_key_iv: "e9f8316aa96d99372e81d90c1c2d0eeb"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.291852 11187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "e8b82b91019a4f43b9b586ee2f58332f"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "37ca7ec389e1619f0f7cae91b20f81f7"
server_key_iv: "e9f8316aa96d99372e81d90c1c2d0eeb"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.294802 11187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:36:51.296757 11202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.297756 11187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:51.297865 11187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "e8b82b91019a4f43b9b586ee2f58332f"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "37ca7ec389e1619f0f7cae91b20f81f7"
server_key_iv: "e9f8316aa96d99372e81d90c1c2d0eeb"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.297950 11187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:51.308878 11187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:51.311489 11187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:51.311648 11187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:51.312109 11187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:51.312754 11187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:51.312794 11187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.312850 11187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:51.312877 11187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.320341 11187 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:37153
I20260923 14:36:51.320348 11315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:37153 every 8 connection(s)
I20260923 14:36:51.321045 11187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
Sep 23 14:36:51 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:51.330318 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 11187
I20260923 14:36:51.330430 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:36:51.330722 2920 external_mini_cluster.cc:1428] Setting key 1de054e9a3cb4bb5255684bb9825abdd
I20260923 14:36:51.331621 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:51.322510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:56631 (local address 127.2.218.62:38085)
0923 14:36:51.322622 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:51.322625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:51.323418 (+ 793us) server_negotiation.cc:407] Connection header received
0923 14:36:51.324128 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:51.324131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:51.324190 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:51.324257 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:51.325140 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.325615 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.326617 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.326822 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.326919 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:51.328773 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:51.328775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:51.328777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:51.328797 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:51.329814 (+ 1017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:51.330304 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:51.330307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:51.330309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:51.330377 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:51.330749 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:51.330751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:51.330752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:51.330935 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:51.331006 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:51.331415 (+ 409us) server_negotiation.cc:299] Negotiation successful
0923 14:36:51.331519 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:36:51.332283 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:51.322795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:56631)
0923 14:36:51.323306 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:51.323337 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:51.323958 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:51.324390 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:51.324396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:51.324667 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:51.325016 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:51.325026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.325724 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.325729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:51.326506 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:51.326514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.326946 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.326950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:51.327089 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:51.327476 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:51.327488 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:51.328635 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:51.329905 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:51.329909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:51.329917 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:51.330194 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:51.330494 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:51.330497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:51.330498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:51.330664 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:51.331027 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:51.331032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:51.331223 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 14:36:51.331350 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 14:36:51.331597 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":94,"threads_started":1}
I20260923 14:36:51.333282 11316 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38085
I20260923 14:36:51.333498 11316 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:51.333864 11316 heartbeater.cc:511] Master 127.2.218.62:38085 requested a full tablet report, sending...
I20260923 14:36:51.335109 11125 ts_manager.cc:194] Registered new tserver with Master: e8b82b91019a4f43b9b586ee2f58332f (127.2.218.1:37153)
I20260923 14:36:51.336073 11125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:56631
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:36:51.352394 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38085
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:51.443684 11321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:51.443920 11321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:51.443979 11321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:51.447458 11321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:51.447520 11321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:51.447587 11321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:51.451994 11321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:51.452836 11321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:51.453706 11321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:51.461887 11326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:51.461895 11327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:51.461902 11329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:51.462283 11321 server_base.cc:1034] running on GCE node
I20260923 14:36:51.462610 11321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:51.463100 11321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:51.464244 11321 hybrid_clock.cc:648] HybridClock initialized: now 1790174211464218 us; error 37 us; skew 500 ppm
Sep 23 14:36:51 dist-test-slave-zl47 krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:51.466614 11321 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:36:51.467275 11321 webserver.cc:460] Webserver started at http://127.2.218.2:35875/ using document root <none> and password file <none>
I20260923 14:36:51.467724 11321 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:51.467801 11321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:51.467983 11321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:51.469573 11321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "74c6bba9ce5a4fc189b882c073913da0"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "3dd9b5b968cb652c14e907d4a0944778"
server_key_iv: "4e6c8517acdd389e232566bf831c998e"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.470129 11321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "74c6bba9ce5a4fc189b882c073913da0"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "3dd9b5b968cb652c14e907d4a0944778"
server_key_iv: "4e6c8517acdd389e232566bf831c998e"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.473124 11321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:36:51.475052 11336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.476022 11321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:51.476133 11321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "74c6bba9ce5a4fc189b882c073913da0"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "3dd9b5b968cb652c14e907d4a0944778"
server_key_iv: "4e6c8517acdd389e232566bf831c998e"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.476221 11321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:51.496058 11321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:51.498950 11321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:51.499104 11321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:51.499675 11321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:51.500358 11321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:51.500399 11321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.500452 11321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:51.500483 11321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.507655 11321 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:44321
I20260923 14:36:51.507678 11449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:44321 every 8 connection(s)
I20260923 14:36:51.508327 11321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
Sep 23 14:36:51 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:51.516923 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 11321
I20260923 14:36:51.517046 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:36:51.517333 2920 external_mini_cluster.cc:1428] Setting key 17f39f9342e14f063ec32dfe8abe6d52
I20260923 14:36:51.519297 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:51.509701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:52851 (local address 127.2.218.62:38085)
0923 14:36:51.509815 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:51.509818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:51.510602 (+ 784us) server_negotiation.cc:407] Connection header received
0923 14:36:51.511400 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:51.511403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:51.511456 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:51.511548 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:51.512438 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.512897 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.513882 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.514080 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.514229 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:51.516120 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:51.516123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:51.516125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:51.516146 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:51.517269 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:51.517729 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:51.517733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:51.517735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:51.517834 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:51.518179 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:51.518181 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:51.518183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:51.518340 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:51.518444 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:51.518961 (+ 517us) server_negotiation.cc:299] Negotiation successful
0923 14:36:51.519075 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:36:51.519755 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:51.509965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:52851)
0923 14:36:51.510484 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:51.510514 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:51.511215 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:51.511689 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:51.511695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:51.511957 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:51.512315 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:51.512325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.513005 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.513010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:51.513768 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:51.513776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.514255 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.514259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:51.514398 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:51.514839 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:51.514852 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:51.515983 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:51.517363 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:51.517370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:51.517380 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:51.517647 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:51.517918 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:51.517920 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:51.517921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:51.518081 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:51.518463 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:51.518470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:51.518703 (+ 233us) client_negotiation.cc:769] Sending connection context
0923 14:36:51.518903 (+ 200us) client_negotiation.cc:240] Negotiation successful
0923 14:36:51.519098 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":113,"threads_started":1}
I20260923 14:36:51.520710 11450 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38085
I20260923 14:36:51.520928 11450 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:51.521378 11450 heartbeater.cc:511] Master 127.2.218.62:38085 requested a full tablet report, sending...
I20260923 14:36:51.522186 11125 ts_manager.cc:194] Registered new tserver with Master: 74c6bba9ce5a4fc189b882c073913da0 (127.2.218.2:44321)
I20260923 14:36:51.522650 11125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:52851
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:36:51.540356 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38085
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:51.632202 11455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:51.632380 11455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:51.632436 11455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:51.635842 11455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:51.635883 11455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:51.635970 11455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:51.640121 11455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:51.640839 11455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:51.641594 11455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:51.649334 11463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:51.649392 11461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:51.649334 11460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:51.649475 11455 server_base.cc:1034] running on GCE node
I20260923 14:36:51.649932 11455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:51.650480 11455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:51.651638 11455 hybrid_clock.cc:648] HybridClock initialized: now 1790174211651623 us; error 33 us; skew 500 ppm
Sep 23 14:36:51 dist-test-slave-zl47 krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:51.653887 11455 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:36:51.654626 11455 webserver.cc:460] Webserver started at http://127.2.218.3:44971/ using document root <none> and password file <none>
I20260923 14:36:51.655099 11455 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:51.655182 11455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:51.655371 11455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:36:51.656937 11455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "ef6d9c30079547b78f3a24a0ab3bf34b"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "8185002226d61405d9a8058c81f9f48a"
server_key_iv: "6ba6d091c8efd6790bebe0d519c4e058"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.657387 11455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "ef6d9c30079547b78f3a24a0ab3bf34b"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "8185002226d61405d9a8058c81f9f48a"
server_key_iv: "6ba6d091c8efd6790bebe0d519c4e058"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.660475 11455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:36:51.662426 11470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.663390 11455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:51.663504 11455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "ef6d9c30079547b78f3a24a0ab3bf34b"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "8185002226d61405d9a8058c81f9f48a"
server_key_iv: "6ba6d091c8efd6790bebe0d519c4e058"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.663592 11455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:51.689970 11455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:51.692610 11455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:51.692768 11455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:51.693259 11455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:51.693923 11455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:51.693990 11455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.694063 11455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:51.694130 11455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.702318 11455 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:33839
I20260923 14:36:51.702346 11583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:33839 every 8 connection(s)
I20260923 14:36:51.703016 11455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:36:51.705057 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 11455
I20260923 14:36:51.705153 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:36:51.705405 2920 external_mini_cluster.cc:1428] Setting key abaf2a080cfc3e2ff3822fa6abd3dea0
Sep 23 14:36:51 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:36:51.714366 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:51.704438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:52673 (local address 127.2.218.62:38085)
0923 14:36:51.704585 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:51.704589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:51.705509 (+ 920us) server_negotiation.cc:407] Connection header received
0923 14:36:51.706569 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:51.706572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:51.706619 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:51.706693 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:51.707853 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.708340 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.709299 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.709483 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.709548 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:51.711586 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:51.711589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:51.711590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:51.711639 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:51.712927 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:51.713309 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:51.713311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:51.713313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:51.713373 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:51.713607 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:51.713609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:51.713609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:51.713743 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:51.713804 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:51.714193 (+ 389us) server_negotiation.cc:299] Negotiation successful
0923 14:36:51.714270 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:36:51.715010 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:51.704736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:52673)
0923 14:36:51.705381 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:51.705413 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:51.706353 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:51.706852 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:51.706860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:51.707275 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:51.707742 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:51.707753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.708435 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.708438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:51.709187 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:51.709195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:51.709595 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:51.709598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:51.709725 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:51.710173 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:51.710189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:51.711458 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:36:51.713017 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:51.713020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:51.713028 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:51.713235 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:51.713439 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:36:51.713440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:36:51.713441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:36:51.713541 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:36:51.713814 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:36:51.713817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:36:51.713983 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 14:36:51.714154 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 14:36:51.714355 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":165,"threads_started":1}
I20260923 14:36:51.715906 11584 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38085
I20260923 14:36:51.716089 11584 heartbeater.cc:464] Registering TS with master...
I20260923 14:36:51.716450 11584 heartbeater.cc:511] Master 127.2.218.62:38085 requested a full tablet report, sending...
I20260923 14:36:51.717265 11125 ts_manager.cc:194] Registered new tserver with Master: ef6d9c30079547b78f3a24a0ab3bf34b (127.2.218.3:33839)
I20260923 14:36:51.717814 11125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:52673
I20260923 14:36:51.718791 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab.
Entry for principal oryx/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab.
WARNING: no policy specified for oryx/127.2.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab.
Entry for principal oryx/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab.
WARNING: no policy specified for oryx/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab.
Entry for principal oryx/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab.
I20260923 14:36:51.778373 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 11187
I20260923 14:36:51.783289 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 11321
I20260923 14:36:51.789238 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 11455
I20260923 14:36:51.793571 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 11093
I20260923 14:36:51.799188 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:38085
--webserver_interface=127.2.218.62
--webserver_port=37139
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:38085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:36:51.893580 11593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:51.893798 11593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:51.893859 11593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:51.897207 11593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:36:51.897285 11593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:51.897322 11593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:51.897377 11593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:36:51.897414 11593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:36:51.901889 11593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:38085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:38085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=37139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:51.902684 11593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:51.903504 11593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:51.910403 11599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:51.910490 11593 server_base.cc:1034] running on GCE node
W20260923 14:36:51.910517 11601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:51.910526 11598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:51.911125 11593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:51.911958 11593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:51.913166 11593 hybrid_clock.cc:648] HybridClock initialized: now 1790174211913146 us; error 41 us; skew 500 ppm
Sep 23 14:36:51 dist-test-slave-zl47 krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:51.915524 11593 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:36:51.916258 11593 webserver.cc:460] Webserver started at http://127.2.218.62:37139/ using document root <none> and password file <none>
I20260923 14:36:51.916708 11593 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:51.916800 11593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:51.919889 11593 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:36:51.921605 11608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:51.922663 11593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:51.922796 11593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "e822827d55f1435baae081a5d5766e09"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "8f4f6e5c8edb51764a1c2f3e2965e6a8"
server_key_iv: "7d1ea2381326c94846fb9453fc816acd"
server_key_version: "encryptionkey@0"
I20260923 14:36:51.923216 11593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:51.938545 11593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:51.941040 11593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:51.941166 11593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:51.947521 11593 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:38085
I20260923 14:36:51.947541 11660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:38085 every 8 connection(s)
I20260923 14:36:51.948346 11593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:36:51.951709 11661 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:36:51.953002 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09: Bootstrap starting.
I20260923 14:36:51.953682 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 11593
I20260923 14:36:51.954070 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:37153
--local_ip_for_outbound_sockets=127.2.218.1
--tserver_master_addrs=127.2.218.62:38085
--webserver_port=37777
--webserver_interface=127.2.218.1
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.218.1
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:36:51.967195 11661 log.cc:824] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09: Log is configured to *not* fsync() on all Append() calls
I20260923 14:36:51.972661 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:36:51.972957 11661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09: Bootstrap complete.
I20260923 14:36:51.976595 11661 raft_consensus.cc:348] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } }
I20260923 14:36:51.976931 11661 raft_consensus.cc:729] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e822827d55f1435baae081a5d5766e09, State: Initialized, Role: FOLLOWER
I20260923 14:36:51.977360 11661 consensus_queue.cc:260] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } }
I20260923 14:36:51.977478 11661 raft_consensus.cc:388] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:36:51.977531 11661 raft_consensus.cc:482] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:36:51.977605 11661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:36:51.979204 11661 raft_consensus.cc:504] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } }
I20260923 14:36:51.979492 11661 leader_election.cc:302] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e822827d55f1435baae081a5d5766e09; no voters:
I20260923 14:36:51.979764 11661 leader_election.cc:288] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 14:36:51.979832 11665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 14:36:51.980031 11665 raft_consensus.cc:686] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [term 2 LEADER]: Becoming Leader. State: Replica: e822827d55f1435baae081a5d5766e09, State: Running, Role: LEADER
I20260923 14:36:51.980325 11665 consensus_queue.cc:237] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } }
I20260923 14:36:51.980777 11661 sys_catalog.cc:564] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:36:51.981110 11665 sys_catalog.cc:455] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e822827d55f1435baae081a5d5766e09. Latest consensus state: current_term: 2 leader_uuid: "e822827d55f1435baae081a5d5766e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } } }
I20260923 14:36:51.981208 11665 sys_catalog.cc:458] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:51.981756 11666 sys_catalog.cc:455] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e822827d55f1435baae081a5d5766e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e822827d55f1435baae081a5d5766e09" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38085 } } }
I20260923 14:36:51.981845 11666 sys_catalog.cc:458] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09 [sys.catalog]: This master's current role is: LEADER
I20260923 14:36:51.982262 11669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:36:51.984949 11669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:36:51.985536 11669 catalog_manager.cc:1259] Loaded cluster ID: a965348fa7714a068bbcbcc463a8abd0
I20260923 14:36:51.985587 11669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:36:51.987550 11669 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:36:51.988178 11669 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e822827d55f1435baae081a5d5766e09: Loaded TSK: 0
I20260923 14:36:51.989207 11669 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:36:52.056685 11663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:52.056896 11663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:52.056929 11663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:52.060284 11663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:52.060328 11663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:52.060398 11663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:36:52.064707 11663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.1.keytab
--principal=oryx/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:37153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=37777
--tserver_master_addrs=127.2.218.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:52.065516 11663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:52.066397 11663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:52.075507 11690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:52.075721 11663 server_base.cc:1034] running on GCE node
W20260923 14:36:52.075502 11688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:52.075502 11687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:52.076216 11663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:52.076718 11663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:52.077885 11663 hybrid_clock.cc:648] HybridClock initialized: now 1790174212077866 us; error 35 us; skew 500 ppm
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174212, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:52.080286 11663 init.cc:377] Logged in from keytab as oryx/127.2.218.1@KRBTEST.COM (short username oryx)
I20260923 14:36:52.081112 11663 webserver.cc:460] Webserver started at http://127.2.218.1:37777/ using document root <none> and password file <none>
I20260923 14:36:52.081650 11663 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:52.081722 11663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:52.085273 11663 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:36:52.087127 11697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:52.088191 11663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:52.088267 11663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "e8b82b91019a4f43b9b586ee2f58332f"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "37ca7ec389e1619f0f7cae91b20f81f7"
server_key_iv: "e9f8316aa96d99372e81d90c1c2d0eeb"
server_key_version: "encryptionkey@0"
I20260923 14:36:52.088728 11663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:52.114665 11663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:52.117281 11663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:52.117408 11663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:52.117827 11663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:52.118503 11663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:52.118544 11663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:52.118578 11663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:52.118602 11663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:52.126328 11663 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:37153
I20260923 14:36:52.126358 11810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:37153 every 8 connection(s)
I20260923 14:36:52.127174 11663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:36:52.129597 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 11663
I20260923 14:36:52.130025 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:44321
--local_ip_for_outbound_sockets=127.2.218.2
--tserver_master_addrs=127.2.218.62:38085
--webserver_port=35875
--webserver_interface=127.2.218.2
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.218.2
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:52.139243 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:52.128909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:51713)
0923 14:36:52.129531 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:52.129580 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:52.130618 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:52.131703 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:52.131709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:52.132002 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:52.132385 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.132397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.134163 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.134166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.134928 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.134938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.135922 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.135925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.136043 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.136540 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:52.136554 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:52.138665 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":112,"threads_started":1}
W20260923 14:36:52.139631 11811 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:38085 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.218.62:38085: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
W20260923 14:36:52.140035 11813 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:52.128777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:51713 (local address 127.2.218.62:38085)
0923 14:36:52.129148 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:52.129184 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:52.129711 (+ 527us) server_negotiation.cc:407] Connection header received
0923 14:36:52.130959 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:52.130969 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:52.131288 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:52.131616 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:52.132554 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.133972 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.135052 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.135770 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.135937 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.138925 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:51713: BlockingRecv error: recv got EOF from 127.2.218.1:51713 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":78,"threads_started":1}
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:52.144697 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:52.140009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:50515)
0923 14:36:52.140308 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:52.140322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:52.140403 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:52.140657 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:52.140660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:52.140808 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:52.140900 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.140906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.141545 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.141551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.142010 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.142017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.142451 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.142455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.142538 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.142951 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:52.142961 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:52.144575 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238}
W20260923 14:36:52.145100 11816 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:52.140058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:50515 (local address 127.2.218.62:38085)
0923 14:36:52.140316 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:52.140326 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:52.140367 (+ 41us) server_negotiation.cc:407] Connection header received
0923 14:36:52.140503 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:52.140506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:52.140550 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:52.140644 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:52.141009 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.141440 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.142133 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.142345 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.142452 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.144691 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:50515: BlockingRecv error: recv got EOF from 127.2.218.1:50515 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":134,"threads_started":1}
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:52.149814 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:52.145209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:54879)
0923 14:36:52.145349 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:52.145363 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:52.145456 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:52.145728 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:52.145732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:52.145882 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:52.145947 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.145953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.146710 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.146717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.147233 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.147241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.147693 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.147697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.147784 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.148229 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:52.148240 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:52.149699 (+ 1459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260923 14:36:52.149953 11811 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:36:52.149896 11816 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:52.145281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:54879 (local address 127.2.218.62:38085)
0923 14:36:52.145394 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:52.145400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:52.145415 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:36:52.145568 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:52.145571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:52.145613 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:52.145714 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:52.146065 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.146583 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.147354 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.147573 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.147695 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.149799 (+ 2104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:54879: BlockingRecv error: recv got EOF from 127.2.218.1:54879 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260923 14:36:52.227223 11815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:52.227439 11815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:52.227497 11815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:52.230979 11815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:52.231045 11815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:52.231119 11815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:36:52.235316 11815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.2.keytab
--principal=oryx/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:44321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=35875
--tserver_master_addrs=127.2.218.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:52.236114 11815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:52.236927 11815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:52.244679 11824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:52.244709 11815 server_base.cc:1034] running on GCE node
W20260923 14:36:52.244679 11822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:52.244709 11821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:52.245229 11815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:52.245761 11815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:52.246922 11815 hybrid_clock.cc:648] HybridClock initialized: now 1790174212246906 us; error 24 us; skew 500 ppm
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174212, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:52.249289 11815 init.cc:377] Logged in from keytab as oryx/127.2.218.2@KRBTEST.COM (short username oryx)
I20260923 14:36:52.249907 11815 webserver.cc:460] Webserver started at http://127.2.218.2:35875/ using document root <none> and password file <none>
I20260923 14:36:52.250365 11815 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:52.250425 11815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:52.253345 11815 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:36:52.255007 11831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:52.255966 11815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260923 14:36:52.256065 11815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "74c6bba9ce5a4fc189b882c073913da0"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "3dd9b5b968cb652c14e907d4a0944778"
server_key_iv: "4e6c8517acdd389e232566bf831c998e"
server_key_version: "encryptionkey@0"
I20260923 14:36:52.256498 11815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:52.265236 11815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:52.267812 11815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:52.267925 11815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:52.268363 11815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:52.268983 11815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:52.269023 11815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:52.269073 11815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:52.269104 11815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:52.276156 11815 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:44321
I20260923 14:36:52.276190 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:44321 every 8 connection(s)
I20260923 14:36:52.276901 11815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:52.285317 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 11815
W20260923 14:36:52.285601 11816 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:52.278458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:35331 (local address 127.2.218.62:38085)
0923 14:36:52.278590 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:52.278594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:52.279432 (+ 838us) server_negotiation.cc:407] Connection header received
0923 14:36:52.280074 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:52.280076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:52.280120 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:52.280184 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:52.281180 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.281623 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.282647 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.282845 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.282919 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.285495 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:35331: BlockingRecv error: recv got EOF from 127.2.218.2:35331 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:36:52.285697 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:33839
--local_ip_for_outbound_sockets=127.2.218.3
--tserver_master_addrs=127.2.218.62:38085
--webserver_port=44971
--webserver_interface=127.2.218.3
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.218.3
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:36:52.286037 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:52.278773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:35331)
0923 14:36:52.279316 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:52.279347 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:52.279930 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:52.280314 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:52.280320 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:52.280577 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:52.281012 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.281024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.281727 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.281729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.282551 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.282559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.282954 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.282956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.283064 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.283501 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:52.283514 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:52.285393 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":99,"threads_started":1}
W20260923 14:36:52.286537 11945 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:38085 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.218.62:38085: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:52.291158 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:52.286898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:35243)
0923 14:36:52.287030 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:52.287039 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:52.287114 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:52.287321 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:52.287324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:52.287446 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:52.287513 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.287517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.288110 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.288115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.288553 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.288559 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.289004 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.289007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.289081 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.289442 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:52.289452 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:52.291050 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260923 14:36:52.291244 11816 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:52.287025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:35243 (local address 127.2.218.62:38085)
0923 14:36:52.287138 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:52.287143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:52.287166 (+ 23us) server_negotiation.cc:407] Connection header received
0923 14:36:52.287206 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:52.287208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:52.287238 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:52.287309 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:52.287608 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.288012 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.288656 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.288908 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.289002 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.291142 (+ 2140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:35243: BlockingRecv error: recv got EOF from 127.2.218.2:35243 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:52.296160 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:52.291552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:60405)
0923 14:36:52.291670 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:52.291685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:52.291766 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:52.291980 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:52.291984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:52.292132 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:52.292228 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.292237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.292889 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.292894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.293760 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.293773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.294136 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.294140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.294234 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.294562 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:52.294576 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:52.296070 (+ 1494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 14:36:52.296262 11816 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:52.291605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:60405 (local address 127.2.218.62:38085)
0923 14:36:52.291694 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:52.291697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:52.291734 (+ 37us) server_negotiation.cc:407] Connection header received
0923 14:36:52.291848 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:52.291856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:52.291885 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:52.291938 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:52.292374 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.292776 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.293866 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.294024 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.294083 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.296155 (+ 2072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:60405: BlockingRecv error: recv got EOF from 127.2.218.2:60405 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260923 14:36:52.296432 11945 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:36:52.378854 11948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:36:52.379037 11948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:36:52.379071 11948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:36:52.382651 11948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:36:52.382694 11948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:36:52.382768 11948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:36:52.387228 11948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/krb5kdc/oryx_127.2.218.3.keytab
--principal=oryx/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:33839
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=44971
--tserver_master_addrs=127.2.218.62:38085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:36:52.387941 11948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:36:52.388693 11948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:36:52.396512 11953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:52.396500 11954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:36:52.396486 11956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:36:52.396840 11948 server_base.cc:1034] running on GCE node
I20260923 14:36:52.397141 11948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:36:52.397686 11948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:36:52.398838 11948 hybrid_clock.cc:648] HybridClock initialized: now 1790174212398812 us; error 35 us; skew 500 ppm
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174212, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:36:52.401335 11948 init.cc:377] Logged in from keytab as oryx/127.2.218.3@KRBTEST.COM (short username oryx)
I20260923 14:36:52.402001 11948 webserver.cc:460] Webserver started at http://127.2.218.3:44971/ using document root <none> and password file <none>
I20260923 14:36:52.402469 11948 fs_manager.cc:359] Metadata directory not provided
I20260923 14:36:52.402536 11948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:36:52.405476 11948 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:36:52.407157 11963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:52.408191 11948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:36:52.408319 11948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "ef6d9c30079547b78f3a24a0ab3bf34b"
format_stamp: "Formatted at 2026-09-23 14:36:51 on dist-test-slave-zl47"
server_key: "8185002226d61405d9a8058c81f9f48a"
server_key_iv: "6ba6d091c8efd6790bebe0d519c4e058"
server_key_version: "encryptionkey@0"
I20260923 14:36:52.408736 11948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:36:52.438022 11948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:36:52.440541 11948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:36:52.440748 11948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:36:52.441221 11948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:36:52.442037 11948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:36:52.442075 11948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:52.442160 11948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:36:52.442184 11948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:36:52.449453 11948 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:33839
I20260923 14:36:52.449482 12076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:33839 every 8 connection(s)
I20260923 14:36:52.450263 11948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:36:52.450917 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 11948
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 14:36:52.461017 11813 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:52.452051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:45113 (local address 127.2.218.62:38085)
0923 14:36:52.452134 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:52.452137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:52.452995 (+ 858us) server_negotiation.cc:407] Connection header received
0923 14:36:52.453858 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:52.453860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:52.453903 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:52.453952 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:52.455289 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.455910 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.457236 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.457473 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.457589 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.460899 (+ 3310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:45113: BlockingRecv error: recv got EOF from 127.2.218.3:45113 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 14:36:52.461126 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:52.451596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.2.218.62:38085)
0923 14:36:52.451735 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:52.451740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:52.452000 (+ 260us) server_negotiation.cc:407] Connection header received
0923 14:36:52.452140 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:52.452144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:52.452189 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:52.452281 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:52.453135 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.453589 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.454811 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.455022 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.455102 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.456710 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:36:52.456714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:36:52.456724 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:36:52.456747 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:36:52.458734 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:52.459209 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:52.459213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:52.459216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:52.459361 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:36:52.460111 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:36:52.460114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:36:52.460115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:36:52.460396 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:36:52.460527 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 14:36:52.460807 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 14:36:52.460969 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:36:52.461563 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:52.452218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:45113)
0923 14:36:52.452848 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:52.452897 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:52.453655 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:52.454147 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:52.454156 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:52.454580 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:52.455147 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.455165 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.456025 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.456030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.457069 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.457078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.457642 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.457646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.457819 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.458360 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:52.458374 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:52.460782 (+ 2408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":86,"threads_started":1}
W20260923 14:36:52.461938 12077 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:38085 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.218.62:38085: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:52.466923 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:52.462332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:48021)
0923 14:36:52.462890 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:52.462903 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:52.462976 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:52.463210 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:52.463213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:52.463381 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:52.463481 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.463489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.464140 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.464142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.464587 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.464594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.464960 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.464962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.465022 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.465348 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:52.465358 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:52.466852 (+ 1494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486}
W20260923 14:36:52.467103 11816 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:52.462425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:48021 (local address 127.2.218.62:38085)
0923 14:36:52.462502 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:52.462505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:52.462960 (+ 455us) server_negotiation.cc:407] Connection header received
0923 14:36:52.463070 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:52.463073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:52.463108 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:52.463179 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:52.463582 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.464048 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.464686 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.464872 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.464938 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.466994 (+ 2056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:48021: BlockingRecv error: recv got EOF from 127.2.218.3:48021 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:52 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:52.471673 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:52.467451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:49847)
0923 14:36:52.467690 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:52.467703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:52.467775 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:52.467978 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:52.467981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:52.468123 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:52.468184 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.468188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.468800 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.468805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.469271 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:52.469278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.469699 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.469703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:52.469772 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.470157 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:52.470169 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:52.471568 (+ 1399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187}
W20260923 14:36:52.471837 12077 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:36:52.471913 11816 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:52.467559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:49847 (local address 127.2.218.62:38085)
0923 14:36:52.467686 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:52.467690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:52.467742 (+ 52us) server_negotiation.cc:407] Connection header received
0923 14:36:52.467852 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:52.467854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:52.467885 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:52.467967 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:52.468285 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.468709 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.469394 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:52.469594 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:52.469700 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:52.471664 (+ 1964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:49847: BlockingRecv error: recv got EOF from 127.2.218.3:49847 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 14:36:53 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:53 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:53.155174 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:53.150475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:43065)
0923 14:36:53.150693 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:53.150714 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:53.150784 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:53.151052 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:53.151054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:53.151209 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:53.151292 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:53.151297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.151929 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.151931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:53.152421 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:53.152428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.152803 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.152805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:53.152860 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:53.153335 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:53.153345 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:53.155076 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
W20260923 14:36:53.155328 12082 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:53.150614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:43065 (local address 127.2.218.62:38085)
0923 14:36:53.150864 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:53.150867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:53.150878 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:36:53.150911 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:53.150913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:53.150948 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:53.151026 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:53.151390 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.151843 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.152511 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.152700 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.152792 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:53.155211 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:43065: BlockingRecv error: recv got EOF from 127.2.218.1:43065 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
Sep 23 14:36:53 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:53 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:53.302332 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:53.296929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:32803)
0923 14:36:53.297137 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:53.297148 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:53.297256 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:53.297496 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:53.297498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:53.297668 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:53.297763 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:53.297767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.298540 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.298544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:53.299119 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:53.299127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.299558 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.299561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:53.299674 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:53.300190 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:53.300200 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:53.302184 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
W20260923 14:36:53.302382 12082 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:53.297008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:32803 (local address 127.2.218.62:38085)
0923 14:36:53.297139 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:53.297142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:53.297212 (+ 70us) server_negotiation.cc:407] Connection header received
0923 14:36:53.297342 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:53.297345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:53.297401 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:53.297472 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:53.297885 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.298438 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.299238 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.299452 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.299556 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:53.302286 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:32803: BlockingRecv error: recv got EOF from 127.2.218.2:32803 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 23 14:36:53 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:53 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:53.477414 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:53.472389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:49205)
0923 14:36:53.472602 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:53.472631 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:53.472741 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:53.472994 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:53.472997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:53.473170 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:53.473289 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:53.473296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.474048 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.474052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:53.474685 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:53.474693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.475131 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.475134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:53.475226 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:53.475665 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:53.475674 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:53.477316 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
W20260923 14:36:53.477583 12082 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:53.472418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:49205 (local address 127.2.218.62:38085)
0923 14:36:53.472532 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:53.472536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:53.472735 (+ 199us) server_negotiation.cc:407] Connection header received
0923 14:36:53.472831 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:53.472834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:53.472883 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:53.472972 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:53.473415 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.473936 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.474796 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:53.475010 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:53.475130 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:53.477452 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:49205: BlockingRecv error: recv got EOF from 127.2.218.3:49205 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 23 14:36:54 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:54 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:54.161254 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:54.156036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:57833)
0923 14:36:54.156270 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:54.156281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:54.156352 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:54.156682 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:54.156684 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:54.156851 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:54.156967 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:54.156972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.157763 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.157769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:54.158383 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:54.158392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.158870 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.158873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:54.158938 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:54.159350 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:54.159360 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:54.161122 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
W20260923 14:36:54.161417 12086 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:54.156157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:57833 (local address 127.2.218.62:38085)
0923 14:36:54.156378 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:54.156384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:54.156396 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:54.156482 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:54.156485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:54.156538 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:54.156641 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:54.157092 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.157632 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.158489 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.158758 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.158854 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:54.161254 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:57833: BlockingRecv error: recv got EOF from 127.2.218.1:57833 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
Sep 23 14:36:54 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:54 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:54.307428 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:54.303003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:52225)
0923 14:36:54.303228 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:54.303239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:54.303316 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:54.303494 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:54.303496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:54.303633 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:54.303718 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:54.303722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.304321 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.304323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:54.304794 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:54.304800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.305214 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.305218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:54.305277 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:54.305691 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:54.305702 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:54.307320 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
W20260923 14:36:54.307529 12086 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:54.303036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:52225 (local address 127.2.218.62:38085)
0923 14:36:54.303150 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:54.303153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:54.303295 (+ 142us) server_negotiation.cc:407] Connection header received
0923 14:36:54.303372 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:54.303374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:54.303412 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:54.303466 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:54.303803 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.304241 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.304879 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.305080 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.305165 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:54.307430 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:52225: BlockingRecv error: recv got EOF from 127.2.218.2:52225 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 14:36:54 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:54 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:54.483120 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:54.478083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:53359)
0923 14:36:54.478355 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:54.478368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:54.478456 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:54.478655 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:54.478656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:54.478799 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:54.478879 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:54.478883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.479521 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.479523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:54.480081 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:54.480088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.480484 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.480486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:54.480547 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:54.481231 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:54.481243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:54.483003 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
W20260923 14:36:54.483186 12086 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:54.478284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:53359 (local address 127.2.218.62:38085)
0923 14:36:54.478393 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:54.478395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:54.478420 (+ 25us) server_negotiation.cc:407] Connection header received
0923 14:36:54.478526 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:54.478528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:54.478566 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:54.478626 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:54.478961 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.479417 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.480188 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:54.480390 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:54.480463 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:54.483105 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:53359: BlockingRecv error: recv got EOF from 127.2.218.3:53359 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 14:36:55 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:55 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:55.166743 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:55.161896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:37091)
0923 14:36:55.162166 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:55.162177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:55.162259 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:55.162470 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:55.162472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:55.162621 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:55.162711 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:55.162716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.163340 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.163342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:55.163818 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:55.163825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.164179 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.164181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:55.164238 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:55.164723 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:55.164734 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:55.166584 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
W20260923 14:36:55.166824 12090 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:55.161941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:37091 (local address 127.2.218.62:38085)
0923 14:36:55.162174 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:55.162177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:55.162236 (+ 59us) server_negotiation.cc:407] Connection header received
0923 14:36:55.162326 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:55.162328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:55.162366 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:55.162432 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:55.162802 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.163255 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.163904 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.164087 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.164150 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:55.166697 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:37091: BlockingRecv error: recv got EOF from 127.2.218.1:37091 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
Sep 23 14:36:55 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:55 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:55.313275 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:55.308206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:50277)
0923 14:36:55.308453 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:55.308468 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:55.308589 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:55.308826 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:55.308828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:55.308997 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:55.309076 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:55.309080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.309740 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.309746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:55.310294 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:55.310301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.310703 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.310707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:55.310779 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:55.311292 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:55.311302 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:55.313182 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":67,"threads_started":1}
W20260923 14:36:55.313436 12090 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:55.308313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:50277 (local address 127.2.218.62:38085)
0923 14:36:55.308453 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:55.308457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:55.308541 (+ 84us) server_negotiation.cc:407] Connection header received
0923 14:36:55.308673 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:55.308677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:55.308721 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:55.308805 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:55.309198 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.309641 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.310396 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.310602 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.310697 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:55.313325 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:50277: BlockingRecv error: recv got EOF from 127.2.218.2:50277 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 14:36:55 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:55 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:55.488607 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:55.483775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:39667)
0923 14:36:55.484062 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:55.484075 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:55.484197 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:55.484457 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:55.484460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:55.484655 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:55.484741 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:55.484747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.485471 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.485476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:55.486013 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:55.486021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.486470 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.486473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:55.486549 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:55.486959 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:55.486969 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:55.488470 (+ 1501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":105,"threads_started":1}
W20260923 14:36:55.488664 12090 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:55.483895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:39667 (local address 127.2.218.62:38085)
0923 14:36:55.484031 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:55.484035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:55.484157 (+ 122us) server_negotiation.cc:407] Connection header received
0923 14:36:55.484301 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:55.484304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:55.484355 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:55.484437 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:55.484852 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.485360 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.486171 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:55.486369 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:55.486467 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:55.488574 (+ 2107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:39667: BlockingRecv error: recv got EOF from 127.2.218.3:39667 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 14:36:56 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:56 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:56.187666 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:56.167607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:46703)
0923 14:36:56.167892 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:56.167907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:56.168004 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:56.168295 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:56.168298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:56.168454 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:56.168543 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:56.168548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.169207 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.169209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:56.169716 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:56.169723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.170090 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.170143 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:56.170252 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:56.185764 (+ 15512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:56.185776 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:56.187525 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
W20260923 14:36:56.187738 12094 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:56.167640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:46703 (local address 127.2.218.62:38085)
0923 14:36:56.167920 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:56.167922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:56.167976 (+ 54us) server_negotiation.cc:407] Connection header received
0923 14:36:56.168097 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:56.168100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:56.168148 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:56.168260 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:56.168630 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.169100 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.169802 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.169997 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.170067 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:56.187631 (+ 17564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:46703: BlockingRecv error: recv got EOF from 127.2.218.1:46703 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":74,"threads_started":1}
Sep 23 14:36:56 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:56 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:56.318207 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:56.313871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:43531)
0923 14:36:56.314079 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:56.314119 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:56.314199 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:56.314394 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:56.314396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:56.314531 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:56.314617 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:56.314621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.315231 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.315237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:56.315711 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:56.315717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.316059 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.316061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:56.316116 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:56.316513 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:56.316524 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:56.318051 (+ 1527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
W20260923 14:36:56.318282 12094 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:56.313902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:43531 (local address 127.2.218.62:38085)
0923 14:36:56.314016 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:56.314019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:56.314180 (+ 161us) server_negotiation.cc:407] Connection header received
0923 14:36:56.314267 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:56.314268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:56.314305 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:56.314363 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:56.314715 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.315150 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.315796 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.315972 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.316036 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:56.318172 (+ 2136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:43531: BlockingRecv error: recv got EOF from 127.2.218.2:43531 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 14:36:56 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:56 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:56.493995 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:56.489394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:52623)
0923 14:36:56.489642 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:56.489655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:56.489758 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:56.490007 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:56.490010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:56.490210 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:56.490297 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:56.490302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.491042 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.491045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:56.491613 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:56.491620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.491975 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.491977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:56.492031 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:56.492428 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:56.492439 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:56.493904 (+ 1465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
W20260923 14:36:56.494140 12094 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:56.489446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:52623 (local address 127.2.218.62:38085)
0923 14:36:56.489568 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:56.489572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:56.489729 (+ 157us) server_negotiation.cc:407] Connection header received
0923 14:36:56.489841 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:56.489844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:56.489894 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:56.489960 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:56.490393 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.490943 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.491715 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:56.491893 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:56.491983 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:56.494007 (+ 2024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:52623: BlockingRecv error: recv got EOF from 127.2.218.3:52623 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 14:36:57 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:57 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:57.193221 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:57.188380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:59565)
0923 14:36:57.188639 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:57.188652 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:57.188729 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:57.188941 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:57.188943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:57.189103 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:57.189205 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:57.189210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.189815 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.189817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:57.190379 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:57.190388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.190737 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.190739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:57.190795 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:57.191228 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:57.191237 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:57.193081 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":66,"threads_started":1}
W20260923 14:36:57.193360 12098 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:57.188424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:59565 (local address 127.2.218.62:38085)
0923 14:36:57.188677 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:57.188680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:57.188691 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:36:57.188803 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:57.188805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:57.188840 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:57.188905 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:57.189294 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.189736 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.190466 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.190656 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.190720 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:57.193245 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:59565: BlockingRecv error: recv got EOF from 127.2.218.1:59565 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
Sep 23 14:36:57 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:57 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:57.323361 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:57.318958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:33103)
0923 14:36:57.319187 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:57.319201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:57.319316 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:57.319497 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:57.319499 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:57.319636 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:57.319728 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:57.319733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.320324 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.320326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:57.320816 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:57.320823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.321160 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.321163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:57.321214 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:57.321597 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:57.321606 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:57.323258 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
W20260923 14:36:57.323491 12098 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:57.319065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:33103 (local address 127.2.218.62:38085)
0923 14:36:57.319185 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:57.319190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:57.319291 (+ 101us) server_negotiation.cc:407] Connection header received
0923 14:36:57.319400 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:57.319402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:57.319439 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:57.319489 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:57.319812 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.320244 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.320908 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.321081 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.321144 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:57.323367 (+ 2223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:33103: BlockingRecv error: recv got EOF from 127.2.218.2:33103 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 23 14:36:57 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:57 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:57.499413 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:57.494865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:59749)
0923 14:36:57.495117 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:57.495128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:57.495208 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:57.495387 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:57.495388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:57.495525 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:57.495602 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:57.495606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.496189 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.496190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:57.496661 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:57.496667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.497056 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.497059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:57.497150 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:57.497651 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:57.497661 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:57.499312 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
W20260923 14:36:57.499509 12098 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:57.494901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:59749 (local address 127.2.218.62:38085)
0923 14:36:57.495016 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:57.495020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:57.495188 (+ 168us) server_negotiation.cc:407] Connection header received
0923 14:36:57.495262 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:57.495264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:57.495300 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:57.495356 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:57.495682 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.496102 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.496749 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:57.496938 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:57.497036 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:57.499417 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:59749: BlockingRecv error: recv got EOF from 127.2.218.3:59749 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 14:36:58 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:58 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:58.198438 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:58.193871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:52705)
0923 14:36:58.194140 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:58.194161 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:58.194242 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:58.194454 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:58.194456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:58.194607 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:58.194703 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:58.194708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.195331 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.195333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:58.195805 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:58.195811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.196223 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.196226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:58.196278 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:58.196744 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:58.196754 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:58.198337 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
W20260923 14:36:58.198621 12102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:58.193921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:52705 (local address 127.2.218.62:38085)
0923 14:36:58.194189 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:58.194192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:58.194204 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:58.194318 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:58.194320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:58.194357 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:58.194422 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:58.194789 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.195252 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.195917 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.196119 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.196212 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:58.198458 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:52705: BlockingRecv error: recv got EOF from 127.2.218.1:52705 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":54,"threads_started":1}
Sep 23 14:36:58 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:58 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:58.328850 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:58.323989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:55105)
0923 14:36:58.324221 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:58.324233 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:58.324305 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:58.324557 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:58.324559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:58.324732 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:58.324812 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:58.324818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.325606 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.325610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:58.326161 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:58.326174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.326658 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.326661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:58.326744 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:58.327155 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:58.327165 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:58.328753 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
W20260923 14:36:58.328949 12102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:58.324082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:55105 (local address 127.2.218.62:38085)
0923 14:36:58.324238 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:58.324245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:58.324271 (+ 26us) server_negotiation.cc:407] Connection header received
0923 14:36:58.324365 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:58.324367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:58.324437 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:58.324523 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:58.324910 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.325492 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.326328 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.326539 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.326647 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:58.328858 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:55105: BlockingRecv error: recv got EOF from 127.2.218.2:55105 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 23 14:36:58 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:58 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:58.504926 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:58.500092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:54487)
0923 14:36:58.500390 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:58.500402 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:58.500518 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:58.500743 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:58.500745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:58.500930 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:58.501007 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:58.501012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.501680 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.501683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:58.502185 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:58.502193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.502597 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.502601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:58.502683 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:58.503118 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:58.503132 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:58.504800 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":80,"threads_started":1}
W20260923 14:36:58.505054 12102 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:58.500213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:54487 (local address 127.2.218.62:38085)
0923 14:36:58.500350 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:58.500353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:58.500477 (+ 124us) server_negotiation.cc:407] Connection header received
0923 14:36:58.500596 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:58.500598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:58.500640 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:58.500721 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:58.501111 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.501569 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.502299 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:58.502497 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:58.502586 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:58.504941 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:54487: BlockingRecv error: recv got EOF from 127.2.218.3:54487 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 14:36:59 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:59 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:59.204499 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:59.199151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:43739)
0923 14:36:59.199441 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:59.199454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:59.199541 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:59.199767 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:59.199769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:59.199948 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:59.200043 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:59.200049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.200802 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.200806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:59.201555 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:59.201566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.202140 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.202144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:59.202226 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:59.202652 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:59.202662 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:59.204399 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
W20260923 14:36:59.204663 12106 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:59.199317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:43739 (local address 127.2.218.62:38085)
0923 14:36:59.199511 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:59.199514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:59.199526 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:36:59.199619 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:59.199620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:59.199658 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:59.199730 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:59.200154 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.200615 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.201689 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.202006 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.202078 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:59.204514 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:43739: BlockingRecv error: recv got EOF from 127.2.218.1:43739 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
Sep 23 14:36:59 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:59 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:59.334247 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:59.329549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:59735)
0923 14:36:59.329845 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:59.329855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:59.329964 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:59.330235 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:59.330238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:59.330401 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:59.330479 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:59.330484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.331123 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.331128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:59.331610 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:59.331617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.332007 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.332011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:59.332084 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:59.332550 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:59.332561 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:59.334064 (+ 1503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
W20260923 14:36:59.334273 12106 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:59.329602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:59735 (local address 127.2.218.62:38085)
0923 14:36:59.329740 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:59.329744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:59.329925 (+ 181us) server_negotiation.cc:407] Connection header received
0923 14:36:59.330053 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:59.330058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:59.330121 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:59.330195 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:59.330583 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.331025 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.331719 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.331910 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.332004 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:59.334186 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:59735: BlockingRecv error: recv got EOF from 127.2.218.2:59735 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 23 14:36:59 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:36:59 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:36:59.510099 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:36:59.505631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:39685)
0923 14:36:59.505950 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 14:36:59.505964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:36:59.506050 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:36:59.506262 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:36:59.506264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:36:59.506412 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:36:59.506495 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:59.506499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.507109 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.507111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:59.507569 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:36:59.507574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.507915 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.507917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:36:59.507966 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:59.508369 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:36:59.508379 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:36:59.509960 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":102,"threads_started":1}
W20260923 14:36:59.510197 12106 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:36:59.505720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:39685 (local address 127.2.218.62:38085)
0923 14:36:59.505810 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 14:36:59.505814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:36:59.506032 (+ 218us) server_negotiation.cc:407] Connection header received
0923 14:36:59.506121 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:36:59.506123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:36:59.506167 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:36:59.506224 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:36:59.506581 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.507031 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.507654 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:36:59.507834 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:36:59.507899 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:36:59.510061 (+ 2162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:39685: BlockingRecv error: recv got EOF from 127.2.218.3:39685 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 23 14:37:00 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:00 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:00.209892 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:00.205294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:48633)
0923 14:37:00.205592 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:00.205606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:00.205696 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:00.205935 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:00.205937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:00.206082 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:00.206225 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:00.206231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.206884 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.206886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:00.207357 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:00.207363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.207743 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.207745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:00.207799 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:00.208273 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:00.208284 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:00.209796 (+ 1512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":110,"threads_started":1}
W20260923 14:37:00.210083 12110 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:00.205431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:48633 (local address 127.2.218.62:38085)
0923 14:37:00.205643 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:00.205646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:00.205661 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:37:00.205787 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:00.205789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:00.205831 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:00.205908 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:00.206322 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.206788 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.207448 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.207649 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.207725 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:00.209938 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:48633: BlockingRecv error: recv got EOF from 127.2.218.1:48633 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":56,"threads_started":1}
Sep 23 14:37:00 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:00 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:00.339501 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:00.334914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:58651)
0923 14:37:00.335116 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:00.335135 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:00.335262 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:00.335490 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:00.335493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:00.335658 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:00.335738 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:00.335743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.336493 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.336496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:00.336986 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:00.336993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.337388 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.337391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:00.337464 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:00.337858 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:00.337867 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:00.339410 (+ 1543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
W20260923 14:37:00.339712 12110 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:00.334995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:58651 (local address 127.2.218.62:38085)
0923 14:37:00.335095 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:00.335099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:00.335220 (+ 121us) server_negotiation.cc:407] Connection header received
0923 14:37:00.335353 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:00.335356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:00.335400 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:00.335470 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:00.335839 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.336399 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.337097 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.337294 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.337386 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:00.339558 (+ 2172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:58651: BlockingRecv error: recv got EOF from 127.2.218.2:58651 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 23 14:37:00 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:00 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:00.515762 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:00.510808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:34067)
0923 14:37:00.511071 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:00.511082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:00.511181 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:00.511374 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:00.511376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:00.511519 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:00.511600 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:00.511604 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.512357 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.512359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:00.512835 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:00.512842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.513197 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.513199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:00.513257 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:00.513655 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:00.513666 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:00.515657 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
W20260923 14:37:00.515856 12110 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:00.510849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:34067 (local address 127.2.218.62:38085)
0923 14:37:00.510969 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:00.510973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:00.511143 (+ 170us) server_negotiation.cc:407] Connection header received
0923 14:37:00.511252 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:00.511254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:00.511295 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:00.511346 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:00.511733 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.512273 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.512921 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:00.513104 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:00.513187 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:00.515759 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:34067: BlockingRecv error: recv got EOF from 127.2.218.3:34067 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 23 14:37:01 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:01 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:01.222291 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:01.210552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:45373)
0923 14:37:01.210802 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:01.210814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:01.210908 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:01.211110 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:01.211113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:01.211282 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:01.211380 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:01.211386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.212033 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.212036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:01.212589 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:01.212600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.212993 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.212995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:01.213055 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:01.219997 (+ 6942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:01.220008 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:01.222143 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
W20260923 14:37:01.222357 12114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:01.210588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:45373 (local address 127.2.218.62:38085)
0923 14:37:01.210825 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:01.210828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:01.210862 (+ 34us) server_negotiation.cc:407] Connection header received
0923 14:37:01.210980 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:01.210982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:01.211019 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:01.211078 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:01.211490 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.211946 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.212706 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.212902 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.212969 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:01.222237 (+ 9268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:45373: BlockingRecv error: recv got EOF from 127.2.218.1:45373 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
Sep 23 14:37:01 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:01 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:01.344478 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:01.340304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:40765)
0923 14:37:01.340514 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:01.340526 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:01.340593 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:01.340777 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:01.340778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:01.340922 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:01.341004 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:01.341008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.341617 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.341620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:01.342122 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:01.342129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.342468 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.342470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:01.342522 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:01.342884 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:01.342894 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:01.344388 (+ 1494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
W20260923 14:37:01.344592 12114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:01.340477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:40765 (local address 127.2.218.62:38085)
0923 14:37:01.340584 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:01.340586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:01.340597 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:37:01.340661 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:01.340663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:01.340697 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:01.340747 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:01.341091 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.341525 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.342204 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.342386 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.342451 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:01.344496 (+ 2045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:40765: BlockingRecv error: recv got EOF from 127.2.218.2:40765 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 14:37:01 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:01 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:01.521118 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:01.516558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:59223)
0923 14:37:01.516796 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:01.516808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:01.516895 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:01.517089 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:01.517091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:01.517236 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:01.517320 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:01.517326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.517918 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.517920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:01.518409 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:01.518416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.518771 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.518773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:01.518830 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:01.519221 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:01.519232 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:01.521026 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
W20260923 14:37:01.521205 12114 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:01.516610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:59223 (local address 127.2.218.62:38085)
0923 14:37:01.516722 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:01.516727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:01.516855 (+ 128us) server_negotiation.cc:407] Connection header received
0923 14:37:01.516961 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:01.516964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:01.516999 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:01.517052 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:01.517415 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.517836 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.518497 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:01.518679 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:01.518745 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:01.521129 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:59223: BlockingRecv error: recv got EOF from 127.2.218.3:59223 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 14:37:02 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:02 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:02.227845 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:02.222974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:37701)
0923 14:37:02.223226 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:02.223238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:02.223326 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:02.223532 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:02.223534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:02.223701 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:02.223794 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:02.223800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.224422 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.224424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:02.224913 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:02.224920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.225351 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.225353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:02.225414 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:02.225937 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:02.225954 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:02.227706 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
W20260923 14:37:02.227906 12118 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:02.223038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:37701 (local address 127.2.218.62:38085)
0923 14:37:02.223238 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:02.223241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:02.223299 (+ 58us) server_negotiation.cc:407] Connection header received
0923 14:37:02.223395 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:02.223397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:02.223432 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:02.223498 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:02.223882 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.224336 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.225002 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.225248 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.225328 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:02.227817 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:37701: BlockingRecv error: recv got EOF from 127.2.218.1:37701 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
Sep 23 14:37:02 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:02 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:02.350230 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:02.345154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:42027)
0923 14:37:02.345535 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:02.345553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:02.345693 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:02.345996 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:02.345999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:02.346211 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:02.346302 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:02.346309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.347055 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.347061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:02.347562 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:02.347569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.348007 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.348011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:02.348092 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:02.348513 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:02.348523 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:02.350061 (+ 1538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":71,"threads_started":1}
W20260923 14:37:02.350339 12118 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:02.345255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:42027 (local address 127.2.218.62:38085)
0923 14:37:02.345486 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:02.345491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:02.345618 (+ 127us) server_negotiation.cc:407] Connection header received
0923 14:37:02.345818 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:02.345823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:02.345882 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:02.345972 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:02.346426 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.346950 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.347699 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.347898 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.348005 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:02.350238 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:42027: BlockingRecv error: recv got EOF from 127.2.218.2:42027 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
Sep 23 14:37:02 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:02 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:02.526739 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:02.521898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:59381)
0923 14:37:02.522179 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:02.522191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:02.522309 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:02.522553 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:02.522557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:02.522739 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:02.522821 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:02.522827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.523487 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.523492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:02.523979 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:02.523986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.524389 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.524392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:02.524467 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:02.524938 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:02.524949 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:02.526631 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
W20260923 14:37:02.526903 12118 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:02.522024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:59381 (local address 127.2.218.62:38085)
0923 14:37:02.522171 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:02.522175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:02.522290 (+ 115us) server_negotiation.cc:407] Connection header received
0923 14:37:02.522396 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:02.522400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:02.522451 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:02.522528 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:02.522927 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.523389 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.524089 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:02.524291 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:02.524384 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:02.526758 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:59381: BlockingRecv error: recv got EOF from 127.2.218.3:59381 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 14:37:03 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:03 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:03.233495 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:03.228729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:34317)
0923 14:37:03.228985 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:03.229005 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:03.229121 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:03.229361 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:03.229363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:03.229533 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:03.229630 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:03.229636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.230298 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.230300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:03.230797 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:03.230805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.231184 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.231187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:03.231243 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:03.231788 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:03.231799 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:03.233369 (+ 1570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
W20260923 14:37:03.233667 12122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:03.228768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:34317 (local address 127.2.218.62:38085)
0923 14:37:03.228981 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:03.228984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:03.229087 (+ 103us) server_negotiation.cc:407] Connection header received
0923 14:37:03.229211 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:03.229214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:03.229253 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:03.229321 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:03.229727 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.230210 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.230889 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.231080 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.231145 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:03.233553 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:34317: BlockingRecv error: recv got EOF from 127.2.218.1:34317 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
Sep 23 14:37:03 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:03 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:03.355672 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:03.350838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:44043)
0923 14:37:03.351130 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:03.351143 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:03.351282 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:03.351522 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:03.351524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:03.351716 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:03.351802 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:03.351807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.352495 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.352501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:03.352996 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:03.353003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.353433 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.353436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:03.353514 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:03.353924 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:03.353933 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:03.355575 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":75,"threads_started":1}
W20260923 14:37:03.355836 12122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:03.350940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:44043 (local address 127.2.218.62:38085)
0923 14:37:03.351083 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:03.351087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:03.351222 (+ 135us) server_negotiation.cc:407] Connection header received
0923 14:37:03.351370 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:03.351374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:03.351417 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:03.351500 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:03.351908 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.352394 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.353111 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.353331 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.353424 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:03.355698 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:44043: BlockingRecv error: recv got EOF from 127.2.218.2:44043 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 14:37:03 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:03 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:03.532414 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:03.527451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:54493)
0923 14:37:03.527851 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:03.527866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:03.527972 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:03.528230 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:03.528233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:03.528424 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:03.528511 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:03.528516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.529211 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.529216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:03.529714 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:03.529721 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.530189 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.530193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:03.530274 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:03.530692 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:03.530703 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:03.532312 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":86,"threads_started":1}
W20260923 14:37:03.532537 12122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:03.527489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:54493 (local address 127.2.218.62:38085)
0923 14:37:03.527630 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:03.527634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:03.527921 (+ 287us) server_negotiation.cc:407] Connection header received
0923 14:37:03.528072 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:03.528075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:03.528124 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:03.528212 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:03.528625 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.529103 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.529829 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:03.530047 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:03.530191 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:03.532433 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:54493: BlockingRecv error: recv got EOF from 127.2.218.3:54493 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 23 14:37:04 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:04 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:04.239671 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:04.234415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:58447)
0923 14:37:04.234727 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:04.234749 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:04.234847 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:04.235118 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:04.235120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:04.235383 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:04.235509 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:04.235515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.236253 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.236256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:04.236759 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:04.236766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.237168 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.237170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:04.237226 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:04.237804 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:04.237816 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:04.239571 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
W20260923 14:37:04.239841 12126 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:04.234506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:58447 (local address 127.2.218.62:38085)
0923 14:37:04.234761 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:04.234766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:04.234798 (+ 32us) server_negotiation.cc:407] Connection header received
0923 14:37:04.234956 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:04.234959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:04.235021 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:04.235097 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:04.235641 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.236153 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.236856 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.237054 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.237151 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:04.239689 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:58447: BlockingRecv error: recv got EOF from 127.2.218.1:58447 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":60,"threads_started":1}
Sep 23 14:37:04 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:04 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:04.361274 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:04.356328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:36151)
0923 14:37:04.356548 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:04.356559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:04.356644 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:04.356830 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:04.356832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:04.356979 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:04.357064 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:04.357068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.357977 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.357980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:04.358576 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:04.358585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.358964 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.358966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:04.359035 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:04.359557 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:04.359576 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:04.361169 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
W20260923 14:37:04.361400 12126 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:04.356409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:36151 (local address 127.2.218.62:38085)
0923 14:37:04.356527 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:04.356530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:04.356615 (+ 85us) server_negotiation.cc:407] Connection header received
0923 14:37:04.356702 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:04.356703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:04.356741 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:04.356796 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:04.357148 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.357849 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.358667 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.358881 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.358950 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:04.361294 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:36151: BlockingRecv error: recv got EOF from 127.2.218.2:36151 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 14:37:04 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:04 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:04.538645 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:04.533235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:35685)
0923 14:37:04.533517 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:04.533529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:04.533645 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:04.533917 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:04.533919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:04.534188 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:04.534271 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:04.534276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.534937 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.534942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:04.535451 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:04.535458 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.535898 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.535901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:04.536006 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:04.536456 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:04.536471 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:04.538465 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":72,"threads_started":1}
W20260923 14:37:04.538810 12126 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:04.533321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:35685 (local address 127.2.218.62:38085)
0923 14:37:04.533438 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:04.533442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:04.533597 (+ 155us) server_negotiation.cc:407] Connection header received
0923 14:37:04.533744 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:04.533746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:04.533794 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:04.533886 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:04.534377 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.534838 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.535566 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:04.535786 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:04.535884 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:04.538677 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:35685: BlockingRecv error: recv got EOF from 127.2.218.3:35685 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 14:37:05 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:05 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:05.245287 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:05.240348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:56425)
0923 14:37:05.240578 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:05.240591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:05.240678 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:05.240916 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:05.240918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:05.241076 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:05.241182 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:05.241187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.241840 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.241842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:05.242360 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:05.242368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.242736 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.242738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:05.242793 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:05.243227 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:05.243237 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:05.245120 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
W20260923 14:37:05.245340 12130 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:05.240392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:56425 (local address 127.2.218.62:38085)
0923 14:37:05.240624 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:05.240627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:05.240640 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:37:05.240756 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:05.240758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:05.240801 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:05.240878 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:05.241275 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.241751 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.242446 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.242640 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.242724 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:05.245226 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:56425: BlockingRecv error: recv got EOF from 127.2.218.1:56425 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
Sep 23 14:37:05 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:05 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:05.366416 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:05.361958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:48281)
0923 14:37:05.362205 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:05.362216 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:05.362289 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:05.362501 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:05.362503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:05.362646 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:05.362740 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:05.362745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.363441 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.363443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:05.363925 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:05.363930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.364280 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.364283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:05.364335 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:05.364745 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:05.364754 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:05.366312 (+ 1558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
W20260923 14:37:05.366536 12130 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:05.362126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:48281 (local address 127.2.218.62:38085)
0923 14:37:05.362250 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:05.362253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:05.362265 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:37:05.362361 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:05.362363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:05.362402 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:05.362465 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:05.362870 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.363345 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.364009 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.364197 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.364265 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:05.366432 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:48281: BlockingRecv error: recv got EOF from 127.2.218.2:48281 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 23 14:37:05 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:05 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:05.544829 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:05.539321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:54555)
0923 14:37:05.539544 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:05.539555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:05.539638 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:05.539840 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:05.539842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:05.539990 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:05.540087 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:05.540096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.540989 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.540992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:05.541826 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:05.541840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.542434 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.542437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:05.542527 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:05.543057 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:05.543073 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:05.544665 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
W20260923 14:37:05.544862 12130 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:05.539386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:54555 (local address 127.2.218.62:38085)
0923 14:37:05.539505 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:05.539513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:05.539615 (+ 102us) server_negotiation.cc:407] Connection header received
0923 14:37:05.539708 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:05.539710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:05.539750 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:05.539808 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:05.540238 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.540866 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.541962 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:05.542305 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:05.542428 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:05.544768 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:54555: BlockingRecv error: recv got EOF from 127.2.218.3:54555 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 14:37:06 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:06 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:06.258188 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:06.245983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:60605)
0923 14:37:06.246240 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:06.246251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:06.246333 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:06.246527 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:06.246529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:06.246685 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:06.246772 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:06.246777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.247439 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.247442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:06.247935 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:06.247942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.248343 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.248345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:06.248401 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:06.256380 (+ 7979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:06.256392 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:06.258024 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
W20260923 14:37:06.258376 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:06.246036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:60605 (local address 127.2.218.62:38085)
0923 14:37:06.246328 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:06.246331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:06.246345 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:37:06.246385 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:06.246387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:06.246425 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:06.246500 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:06.246863 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.247345 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.248039 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.248257 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.248330 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:06.258233 (+ 9903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:60605: BlockingRecv error: recv got EOF from 127.2.218.1:60605 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":62,"threads_started":1}
Sep 23 14:37:06 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:06 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:06.371773 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:06.367138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:34527)
0923 14:37:06.367390 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:06.367401 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:06.367517 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:06.367761 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:06.367763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:06.367942 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:06.368023 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:06.368028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.368696 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.368698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:06.369191 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:06.369198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.369607 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.369610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:06.369686 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:06.370138 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:06.370155 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:06.371677 (+ 1522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
W20260923 14:37:06.371881 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:06.367191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:34527 (local address 127.2.218.62:38085)
0923 14:37:06.367340 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:06.367343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:06.367466 (+ 123us) server_negotiation.cc:407] Connection header received
0923 14:37:06.367602 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:06.367606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:06.367654 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:06.367740 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:06.368129 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.368591 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.369299 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.369498 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.369592 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:06.371780 (+ 2188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:34527: BlockingRecv error: recv got EOF from 127.2.218.2:34527 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 14:37:06 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:06 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.3@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:06.551167 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:06.545569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.3:55323)
0923 14:37:06.546065 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:06.546078 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:06.546203 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:06.546430 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:06.546433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:06.546634 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:06.546768 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:06.546778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.547942 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.547945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:06.548439 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:06.548445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.548873 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.548876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:06.548952 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:06.549412 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:06.549424 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:06.551069 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":79,"threads_started":1}
W20260923 14:37:06.551362 12134 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:06.545632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:55323 (local address 127.2.218.62:38085)
0923 14:37:06.545842 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:06.545848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:06.546156 (+ 308us) server_negotiation.cc:407] Connection header received
0923 14:37:06.546285 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:06.546287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:06.546331 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:06.546410 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:06.547389 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.547837 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.548554 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:06.548770 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:06.548863 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:06.551219 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.3:55323: BlockingRecv error: recv got EOF from 127.2.218.3:55323 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 23 14:37:07 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:07 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.1@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:07.264021 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:07.259146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.1:46717)
0923 14:37:07.259399 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:07.259411 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:07.259513 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:07.259766 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:07.259769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:07.259956 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:07.260038 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:07.260043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.260749 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.260753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:07.261278 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:07.261286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.261707 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.261710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:07.261792 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:07.262306 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:07.262316 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:07.263916 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":77,"threads_started":1}
W20260923 14:37:07.264153 12138 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:07.259188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:46717 (local address 127.2.218.62:38085)
0923 14:37:07.259430 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:07.259435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:07.259465 (+ 30us) server_negotiation.cc:407] Connection header received
0923 14:37:07.259600 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:07.259603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:07.259649 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:07.259739 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:07.260143 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.260641 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.261387 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.261601 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.261695 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:07.264035 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.1:46717: BlockingRecv error: recv got EOF from 127.2.218.1:46717 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":65,"threads_started":1}
Sep 23 14:37:07 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 14:37:07 dist-test-slave-zl47 krb5kdc[11080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.218.2@KRBTEST.COM for oryx/127.2.218.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 14:37:07.377321 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:07.372531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38085 (local address 127.2.218.2:58539)
0923 14:37:07.372772 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:07.372786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:07.372877 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:07.373154 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:07.373157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:07.373372 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:07.373462 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:07.373467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.374242 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.374247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:07.374758 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:07.374765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.375177 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.375180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:07.375263 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:07.375666 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:07.375676 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:07.377171 (+ 1495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.218.62:38085: Server oryx/127.2.218.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
W20260923 14:37:07.377374 12138 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:07.372627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:58539 (local address 127.2.218.62:38085)
0923 14:37:07.372766 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:07.372771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:07.372835 (+ 64us) server_negotiation.cc:407] Connection header received
0923 14:37:07.372976 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:07.372980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:07.373026 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:07.373130 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:07.373580 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.374149 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.374868 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.375078 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.375174 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:07.377282 (+ 2108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.218.2:58539: BlockingRecv error: recv got EOF from 127.2.218.2:58539 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:37:07.459314 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 11663
I20260923 14:37:07.464099 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 11815
I20260923 14:37:07.468690 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 11948
I20260923 14:37:07.472958 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 11593
2026-09-23T14:37:07Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17545 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T14:37:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:37:07Z Disabled control of system clock
I20260923 14:37:07.500540 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:37981
--webserver_interface=127.2.218.62
--webserver_port=0
--builtin_ntp_servers=127.2.218.20:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:37981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 14:37:07.592752 12143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:07.592940 12143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:07.592968 12143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:07.596283 12143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:37:07.596324 12143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:07.596338 12143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:07.596351 12143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:37:07.596364 12143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:37:07.600230 12143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:37981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:37981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:07.601017 12143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:07.602058 12143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:07.609246 12151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:07.609227 12149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:07.609289 12148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:07.609378 12143 server_base.cc:1034] running on GCE node
I20260923 14:37:07.610035 12143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:07.610992 12143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:07.612169 12143 hybrid_clock.cc:648] HybridClock initialized: now 1790174227612137 us; error 48 us; skew 500 ppm
I20260923 14:37:07.613821 12143 webserver.cc:460] Webserver started at http://127.2.218.62:43253/ using document root <none> and password file <none>
I20260923 14:37:07.614389 12143 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:07.614452 12143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:07.614668 12143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:07.616393 12143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "ef3c58f12c9a4390aace4f9604b26a85"
format_stamp: "Formatted at 2026-09-23 14:37:07 on dist-test-slave-zl47"
server_key: "18fff523622b408d17ac92019cf7bde0"
server_key_iv: "b1fb8d4774341fe072d1b3cd490dcebe"
server_key_version: "encryptionkey@0"
I20260923 14:37:07.616935 12143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "ef3c58f12c9a4390aace4f9604b26a85"
format_stamp: "Formatted at 2026-09-23 14:37:07 on dist-test-slave-zl47"
server_key: "18fff523622b408d17ac92019cf7bde0"
server_key_iv: "b1fb8d4774341fe072d1b3cd490dcebe"
server_key_version: "encryptionkey@0"
I20260923 14:37:07.620514 12143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 14:37:07.622650 12157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:07.623672 12143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:37:07.623803 12143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "ef3c58f12c9a4390aace4f9604b26a85"
format_stamp: "Formatted at 2026-09-23 14:37:07 on dist-test-slave-zl47"
server_key: "18fff523622b408d17ac92019cf7bde0"
server_key_iv: "b1fb8d4774341fe072d1b3cd490dcebe"
server_key_version: "encryptionkey@0"
I20260923 14:37:07.623907 12143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:07.645005 12143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:07.645567 12143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:07.645754 12143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:07.652077 12209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:37981 every 8 connection(s)
I20260923 14:37:07.652076 12143 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:37981
I20260923 14:37:07.652952 12143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:37:07.654793 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 12143
I20260923 14:37:07.654912 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:37:07.655205 2920 external_mini_cluster.cc:1428] Setting key 32d5df0948016aa73d86b82bb6dd97ca
I20260923 14:37:07.655618 12210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:37:07.661075 12210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85: Bootstrap starting.
I20260923 14:37:07.663048 12210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85: Neither blocks nor log segments found. Creating new log.
I20260923 14:37:07.663754 12210 log.cc:824] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85: Log is configured to *not* fsync() on all Append() calls
I20260923 14:37:07.663940 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:07.656508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.2.218.62:37981)
0923 14:37:07.657055 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:07.657069 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:07.657101 (+ 32us) server_negotiation.cc:407] Connection header received
0923 14:37:07.657933 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:07.657949 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:07.658350 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:07.658647 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:07.659030 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.659929 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.660827 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.661377 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.661554 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:07.662427 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:07.662432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:07.662442 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:37:07.662770 (+ 328us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:07.662821 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:37:07.662841 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:07.662950 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:07.663092 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 14:37:07.663318 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":118,"threads_started":1}
I20260923 14:37:07.665346 12210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85: No bootstrap required, opened a new log
I20260923 14:37:07.668478 12210 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c58f12c9a4390aace4f9604b26a85" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37981 } }
I20260923 14:37:07.668669 12210 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:37:07.668744 12210 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef3c58f12c9a4390aace4f9604b26a85, State: Initialized, Role: FOLLOWER
I20260923 14:37:07.669111 12210 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c58f12c9a4390aace4f9604b26a85" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37981 } }
I20260923 14:37:07.669256 12210 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:37:07.669317 12210 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:37:07.669397 12210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:37:07.670187 12210 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c58f12c9a4390aace4f9604b26a85" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37981 } }
I20260923 14:37:07.670460 12210 leader_election.cc:302] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef3c58f12c9a4390aace4f9604b26a85; no voters:
I20260923 14:37:07.670698 12210 leader_election.cc:288] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:37:07.670810 12215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:37:07.671080 12215 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [term 1 LEADER]: Becoming Leader. State: Replica: ef3c58f12c9a4390aace4f9604b26a85, State: Running, Role: LEADER
I20260923 14:37:07.671379 12215 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c58f12c9a4390aace4f9604b26a85" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37981 } }
I20260923 14:37:07.671782 12210 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:37:07.673071 12216 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef3c58f12c9a4390aace4f9604b26a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c58f12c9a4390aace4f9604b26a85" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37981 } } }
I20260923 14:37:07.673195 12216 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [sys.catalog]: This master's current role is: LEADER
I20260923 14:37:07.673518 12217 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef3c58f12c9a4390aace4f9604b26a85. Latest consensus state: current_term: 1 leader_uuid: "ef3c58f12c9a4390aace4f9604b26a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef3c58f12c9a4390aace4f9604b26a85" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37981 } } }
I20260923 14:37:07.673640 12217 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85 [sys.catalog]: This master's current role is: LEADER
I20260923 14:37:07.673910 12226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:37:07.676095 12226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:37:07.680830 12226 catalog_manager.cc:1347] Generated new cluster ID: 29f05a1676084bc39c21cfdcd331e72f
I20260923 14:37:07.680886 12226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:37:07.695993 12226 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:37:07.696884 12226 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:37:07.702948 12226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef3c58f12c9a4390aace4f9604b26a85: Generated new TSK 0
I20260923 14:37:07.703528 12226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:37:07.707542 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37981
--builtin_ntp_servers=127.2.218.20:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 14:37:07.806424 12234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:07.806619 12234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:07.806654 12234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:07.809999 12234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:07.810046 12234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:07.810137 12234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:37:07.813998 12234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:07.814792 12234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:07.815842 12234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:37:07.823770 12234 server_base.cc:1034] running on GCE node
W20260923 14:37:07.823728 12240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:07.823728 12239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:07.823734 12242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:07.824311 12234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:07.824849 12234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:07.826025 12234 hybrid_clock.cc:648] HybridClock initialized: now 1790174227826011 us; error 29 us; skew 500 ppm
I20260923 14:37:07.827661 12234 webserver.cc:460] Webserver started at http://127.2.218.1:46413/ using document root <none> and password file <none>
I20260923 14:37:07.828171 12234 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:07.828258 12234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:07.828472 12234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:07.830197 12234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "c0bfc18df19b4132b72024bda9314ad0"
format_stamp: "Formatted at 2026-09-23 14:37:07 on dist-test-slave-zl47"
server_key: "1626372baddf223e11a6ab2ed94d8bd1"
server_key_iv: "dc1cda23dbf70f52d360c35b536b3975"
server_key_version: "encryptionkey@0"
I20260923 14:37:07.830695 12234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "c0bfc18df19b4132b72024bda9314ad0"
format_stamp: "Formatted at 2026-09-23 14:37:07 on dist-test-slave-zl47"
server_key: "1626372baddf223e11a6ab2ed94d8bd1"
server_key_iv: "dc1cda23dbf70f52d360c35b536b3975"
server_key_version: "encryptionkey@0"
I20260923 14:37:07.833813 12234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:37:07.835887 12248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:07.836877 12234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:37:07.836997 12234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "c0bfc18df19b4132b72024bda9314ad0"
format_stamp: "Formatted at 2026-09-23 14:37:07 on dist-test-slave-zl47"
server_key: "1626372baddf223e11a6ab2ed94d8bd1"
server_key_iv: "dc1cda23dbf70f52d360c35b536b3975"
server_key_version: "encryptionkey@0"
I20260923 14:37:07.837095 12234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:07.849718 12234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:07.850301 12234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:07.850469 12234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:07.850996 12234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:07.851765 12234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:07.851840 12234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:07.851898 12234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:07.851949 12234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:07.859962 12234 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:43159
I20260923 14:37:07.859982 12361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:43159 every 8 connection(s)
I20260923 14:37:07.860654 12234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:37:07.868228 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:07.862080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:51955 (local address 127.2.218.62:37981)
0923 14:37:07.862228 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:07.862232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:07.863065 (+ 833us) server_negotiation.cc:407] Connection header received
0923 14:37:07.864178 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:07.864181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:07.864241 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:07.864328 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:07.865078 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.865575 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.866546 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.866746 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.866844 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:07.867754 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:07.867757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:07.867759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:37:07.867771 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:07.867790 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:37:07.867804 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:07.867875 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:07.868015 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 14:37:07.868084 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 14:37:07.868819 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:07.862416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:37981 (local address 127.2.218.1:51955)
0923 14:37:07.862943 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:07.862974 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:07.863991 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:07.864488 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:07.864494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:07.864564 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:07.864945 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:07.864955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.865686 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.865689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:07.866430 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:07.866438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:07.866874 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:07.866877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:07.867013 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:07.867530 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:37:07.867532 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:07.867563 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:37:07.867614 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:07.867877 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:07.867887 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:07.867896 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 14:37:07.868013 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 14:37:07.868239 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":97,"threads_started":1}
I20260923 14:37:07.869531 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 12234
I20260923 14:37:07.869666 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:37:07.869908 12362 heartbeater.cc:347] Connected to a master server at 127.2.218.62:37981
I20260923 14:37:07.869973 2920 external_mini_cluster.cc:1428] Setting key 3c0c1d0187f508143b8c8104f367a1fb
I20260923 14:37:07.870126 12362 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:07.870597 12362 heartbeater.cc:511] Master 127.2.218.62:37981 requested a full tablet report, sending...
I20260923 14:37:07.871892 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37981
--builtin_ntp_servers=127.2.218.20:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 14:37:07.871942 12174 ts_manager.cc:194] Registered new tserver with Master: c0bfc18df19b4132b72024bda9314ad0 (127.2.218.1:43159)
I20260923 14:37:07.873050 12174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.1:51955
W20260923 14:37:07.965672 12365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:07.965909 12365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:07.965946 12365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:07.969360 12365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:07.969409 12365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:07.969452 12365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:37:07.973703 12365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:07.974658 12365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:07.975795 12365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:07.984079 12373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:07.984098 12370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:07.984072 12371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:07.984325 12365 server_base.cc:1034] running on GCE node
I20260923 14:37:07.984655 12365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:07.985189 12365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:07.986353 12365 hybrid_clock.cc:648] HybridClock initialized: now 1790174227986343 us; error 22 us; skew 500 ppm
I20260923 14:37:07.987944 12365 webserver.cc:460] Webserver started at http://127.2.218.2:39893/ using document root <none> and password file <none>
I20260923 14:37:07.988566 12365 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:07.988629 12365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:07.988858 12365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:07.990655 12365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "a628f49f39e24625ae5d3480c05a1802"
format_stamp: "Formatted at 2026-09-23 14:37:07 on dist-test-slave-zl47"
server_key: "6f94fca57a46eaef837ac2d4ab05c69d"
server_key_iv: "eef7a276773ccfa6b14a054bbdd985de"
server_key_version: "encryptionkey@0"
I20260923 14:37:07.991150 12365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "a628f49f39e24625ae5d3480c05a1802"
format_stamp: "Formatted at 2026-09-23 14:37:07 on dist-test-slave-zl47"
server_key: "6f94fca57a46eaef837ac2d4ab05c69d"
server_key_iv: "eef7a276773ccfa6b14a054bbdd985de"
server_key_version: "encryptionkey@0"
I20260923 14:37:07.994247 12365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:37:07.996445 12379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:07.997498 12365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:37:07.997615 12365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "a628f49f39e24625ae5d3480c05a1802"
format_stamp: "Formatted at 2026-09-23 14:37:07 on dist-test-slave-zl47"
server_key: "6f94fca57a46eaef837ac2d4ab05c69d"
server_key_iv: "eef7a276773ccfa6b14a054bbdd985de"
server_key_version: "encryptionkey@0"
I20260923 14:37:07.997712 12365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:08.008948 12365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:08.009479 12365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:08.009639 12365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:08.010133 12365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:08.010910 12365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:08.010970 12365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.011027 12365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:08.011056 12365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.018662 12365 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:46065
I20260923 14:37:08.018728 12492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:46065 every 8 connection(s)
I20260923 14:37:08.019520 12365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:37:08.027076 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 12365
I20260923 14:37:08.027192 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:37:08.027504 2920 external_mini_cluster.cc:1428] Setting key 45bed68f506cc0c5a950e8fe812fecb7
I20260923 14:37:08.028152 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.021149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:56191 (local address 127.2.218.62:37981)
0923 14:37:08.021281 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:08.021284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:08.022146 (+ 862us) server_negotiation.cc:407] Connection header received
0923 14:37:08.023516 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:08.023519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:08.023574 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:08.023674 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:08.024621 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:08.025355 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:08.026371 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:08.026555 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:08.026628 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:08.027652 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:08.027655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:08.027658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:37:08.027669 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:08.027697 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:37:08.027705 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:08.027775 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:08.027955 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 14:37:08.028022 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 14:37:08.028930 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.021463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:37981 (local address 127.2.218.2:56191)
0923 14:37:08.021981 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:08.022013 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:08.023314 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:08.023824 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:08.023829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:08.023899 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:08.024466 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:08.024487 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:08.025474 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:08.025476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:08.026272 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:08.026281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:08.026677 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:08.026681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:08.026795 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:08.027363 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:37:08.027365 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:08.027397 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:37:08.027447 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:08.027787 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:08.027796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:08.027800 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:37:08.027935 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 14:37:08.028175 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
I20260923 14:37:08.029386 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37981
--builtin_ntp_servers=127.2.218.20:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 14:37:08.029949 12493 heartbeater.cc:347] Connected to a master server at 127.2.218.62:37981
I20260923 14:37:08.030161 12493 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:08.030579 12493 heartbeater.cc:511] Master 127.2.218.62:37981 requested a full tablet report, sending...
I20260923 14:37:08.031420 12174 ts_manager.cc:194] Registered new tserver with Master: a628f49f39e24625ae5d3480c05a1802 (127.2.218.2:46065)
I20260923 14:37:08.031912 12174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.2:56191
W20260923 14:37:08.125115 12496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:08.125309 12496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:08.125339 12496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:08.128863 12496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:08.128903 12496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:08.128974 12496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:37:08.132771 12496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:08.133540 12496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:08.134656 12496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:08.142472 12502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:08.142498 12501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:08.142498 12504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:08.142597 12496 server_base.cc:1034] running on GCE node
I20260923 14:37:08.142968 12496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:08.143504 12496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:08.144683 12496 hybrid_clock.cc:648] HybridClock initialized: now 1790174228144663 us; error 37 us; skew 500 ppm
I20260923 14:37:08.146374 12496 webserver.cc:460] Webserver started at http://127.2.218.3:43795/ using document root <none> and password file <none>
I20260923 14:37:08.146844 12496 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:08.146930 12496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:08.147127 12496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:08.148772 12496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "d055997767ae4c959e054b379f3f1fb8"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "4c4622dfd439eb5ad84ea0a4f053422c"
server_key_iv: "7ae44d68c753503f527f1cdbccce4069"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.149269 12496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "d055997767ae4c959e054b379f3f1fb8"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "4c4622dfd439eb5ad84ea0a4f053422c"
server_key_iv: "7ae44d68c753503f527f1cdbccce4069"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.152412 12496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:37:08.154482 12510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.155452 12496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:37:08.155573 12496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "d055997767ae4c959e054b379f3f1fb8"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "4c4622dfd439eb5ad84ea0a4f053422c"
server_key_iv: "7ae44d68c753503f527f1cdbccce4069"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.155678 12496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:08.172317 12496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:08.172957 12496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:08.173159 12496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:08.173774 12496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:08.174757 12496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:08.174798 12496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.174853 12496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:08.174880 12496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.182716 12496 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:44017
I20260923 14:37:08.182737 12623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:44017 every 8 connection(s)
I20260923 14:37:08.183439 12496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:37:08.184522 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 12496
I20260923 14:37:08.184620 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:37:08.184860 2920 external_mini_cluster.cc:1428] Setting key 666c08f5fe13c170f2648a8eda796806
I20260923 14:37:08.191138 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.185115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:44349 (local address 127.2.218.62:37981)
0923 14:37:08.185233 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:08.185236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:08.186214 (+ 978us) server_negotiation.cc:407] Connection header received
0923 14:37:08.187462 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:08.187464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:08.187504 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:08.187561 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:08.188275 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:08.188726 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:08.189652 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:08.189836 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:08.189903 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:08.190795 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:08.190797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:08.190799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:37:08.190809 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:08.190824 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:37:08.190831 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:08.190881 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:08.191003 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 14:37:08.191044 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:37:08.191864 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.185461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:37981 (local address 127.2.218.3:44349)
0923 14:37:08.186063 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:08.186107 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:08.187307 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:08.187692 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:08.187698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:08.187767 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:08.188148 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:08.188157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:08.188821 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:08.188824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:08.189562 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:08.189570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:08.189941 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:08.189943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:08.190055 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:08.190580 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:37:08.190582 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:08.190614 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:37:08.190665 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:08.190900 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:08.190906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:08.190910 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:37:08.190985 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 14:37:08.191157 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
I20260923 14:37:08.192809 12624 heartbeater.cc:347] Connected to a master server at 127.2.218.62:37981
I20260923 14:37:08.193011 12624 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:08.193408 12624 heartbeater.cc:511] Master 127.2.218.62:37981 requested a full tablet report, sending...
I20260923 14:37:08.194311 12174 ts_manager.cc:194] Registered new tserver with Master: d055997767ae4c959e054b379f3f1fb8 (127.2.218.3:44017)
I20260923 14:37:08.194747 12174 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.3:44349
I20260923 14:37:08.198413 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 14:37:08.200888 12213 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:08.200069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.2.218.62:37981)
0923 14:37:08.200160 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:08.200163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:08.200380 (+ 217us) server_negotiation.cc:407] Connection header received
0923 14:37:08.200463 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:08.200464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:08.200493 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:08.200550 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:08.200777 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56238: BlockingRecv error: recv got EOF from 127.0.0.1:56238 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:37:08.201038 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 12234
I20260923 14:37:08.206441 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 12365
I20260923 14:37:08.211018 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 12496
I20260923 14:37:08.215317 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 12143
2026-09-23T14:37:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (740 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T14:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:37:08Z Disabled control of system clock
I20260923 14:37:08.241374 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36337
--webserver_interface=127.2.218.62
--webserver_port=0
--builtin_ntp_servers=127.2.218.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:36337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 14:37:08.333889 12635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:08.334116 12635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:08.334158 12635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:08.337495 12635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:37:08.337543 12635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:08.337559 12635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:08.337575 12635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:37:08.337590 12635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:37:08.341436 12635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:36337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:08.342180 12635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:08.343169 12635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:08.350076 12643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:08.350073 12640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:08.350157 12635 server_base.cc:1034] running on GCE node
W20260923 14:37:08.350325 12641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:08.350805 12635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:08.351733 12635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:08.352883 12635 hybrid_clock.cc:648] HybridClock initialized: now 1790174228352867 us; error 28 us; skew 500 ppm
I20260923 14:37:08.354519 12635 webserver.cc:460] Webserver started at http://127.2.218.62:45969/ using document root <none> and password file <none>
I20260923 14:37:08.355001 12635 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:08.355054 12635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:08.355235 12635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:08.356840 12635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "6fbf831cb0116ce0273b1975b8f2e743"
server_key_iv: "8adcfe34cdf0f0cf0f225268b3123304"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.357332 12635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "6fbf831cb0116ce0273b1975b8f2e743"
server_key_iv: "8adcfe34cdf0f0cf0f225268b3123304"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.360318 12635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:37:08.362247 12649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.363214 12635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:37:08.363344 12635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "6fbf831cb0116ce0273b1975b8f2e743"
server_key_iv: "8adcfe34cdf0f0cf0f225268b3123304"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.363438 12635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:08.367115 12635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:08.367619 12635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:08.367784 12635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:08.373351 12635 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:36337
I20260923 14:37:08.373384 12701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:36337 every 8 connection(s)
I20260923 14:37:08.374186 12635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:37:08.375497 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 12635
I20260923 14:37:08.375613 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:37:08.375900 2920 external_mini_cluster.cc:1428] Setting key 4595a9369a3b46ca0d11335f92d8cd69
I20260923 14:37:08.376998 12702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:37:08.380563 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.377080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.2.218.62:36337)
0923 14:37:08.377629 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:08.377641 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:08.377670 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:37:08.378287 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:08.378302 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:08.378582 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:08.378901 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:08.379579 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:08.379583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:08.379592 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:37:08.379601 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:08.379646 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:37:08.379659 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:08.379789 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:08.379851 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 14:37:08.380044 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":123,"threads_started":1}
I20260923 14:37:08.382002 12702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732: Bootstrap starting.
I20260923 14:37:08.383839 12702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732: Neither blocks nor log segments found. Creating new log.
I20260923 14:37:08.384454 12702 log.cc:824] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732: Log is configured to *not* fsync() on all Append() calls
I20260923 14:37:08.386189 12702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732: No bootstrap required, opened a new log
I20260923 14:37:08.388782 12702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36337 } }
I20260923 14:37:08.388942 12702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:37:08.388978 12702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0dfb3dc2aa4a54ab12736a83f3f732, State: Initialized, Role: FOLLOWER
I20260923 14:37:08.389247 12702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36337 } }
I20260923 14:37:08.389334 12702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:37:08.389371 12702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:37:08.389426 12702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:37:08.390079 12702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36337 } }
I20260923 14:37:08.390362 12702 leader_election.cc:302] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f0dfb3dc2aa4a54ab12736a83f3f732; no voters:
I20260923 14:37:08.390561 12702 leader_election.cc:288] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:37:08.390655 12707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:37:08.390857 12707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [term 1 LEADER]: Becoming Leader. State: Replica: 2f0dfb3dc2aa4a54ab12736a83f3f732, State: Running, Role: LEADER
I20260923 14:37:08.391113 12707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36337 } }
I20260923 14:37:08.391525 12702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:37:08.392544 12708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36337 } } }
I20260923 14:37:08.392573 12709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f0dfb3dc2aa4a54ab12736a83f3f732. Latest consensus state: current_term: 1 leader_uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dfb3dc2aa4a54ab12736a83f3f732" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36337 } } }
I20260923 14:37:08.392645 12708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [sys.catalog]: This master's current role is: LEADER
I20260923 14:37:08.392659 12709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732 [sys.catalog]: This master's current role is: LEADER
I20260923 14:37:08.392930 12716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:37:08.395473 12716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:37:08.399968 12716 catalog_manager.cc:1347] Generated new cluster ID: 6491a82734834eb5aba33fa19fc2b287
I20260923 14:37:08.400022 12716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:37:08.409586 12716 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:37:08.409848 12716 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:37:08.415593 12716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f0dfb3dc2aa4a54ab12736a83f3f732: Generated new TSK 0
I20260923 14:37:08.416071 12716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:37:08.425554 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36337
--builtin_ntp_servers=127.2.218.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 14:37:08.517464 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:08.517647 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:08.517678 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:08.521106 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:08.521145 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:08.521222 12726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:37:08.525151 12726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:08.525913 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:08.527050 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:08.534853 12731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:08.534889 12732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:08.534839 12734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:08.535010 12726 server_base.cc:1034] running on GCE node
I20260923 14:37:08.535461 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:08.535982 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:08.537164 12726 hybrid_clock.cc:648] HybridClock initialized: now 1790174228537148 us; error 27 us; skew 500 ppm
I20260923 14:37:08.538631 12726 webserver.cc:460] Webserver started at http://127.2.218.1:35475/ using document root <none> and password file <none>
I20260923 14:37:08.539088 12726 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:08.539139 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:08.539353 12726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:08.541090 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "64b995d69ced4dcaa0663388456368ba"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "2d7e8f8cc3431cc581a656c6319fde1b"
server_key_iv: "82b3ae704fa2ec0adab65bc0c107a929"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.541584 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "64b995d69ced4dcaa0663388456368ba"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "2d7e8f8cc3431cc581a656c6319fde1b"
server_key_iv: "82b3ae704fa2ec0adab65bc0c107a929"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.544528 12726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:37:08.546665 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.547642 12726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:37:08.547765 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "64b995d69ced4dcaa0663388456368ba"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "2d7e8f8cc3431cc581a656c6319fde1b"
server_key_iv: "82b3ae704fa2ec0adab65bc0c107a929"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.547857 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:08.552577 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:08.553059 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:08.553216 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:08.553653 12726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:08.554421 12726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:08.554462 12726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.554517 12726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:08.554543 12726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.562280 12726 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:43923
I20260923 14:37:08.562346 12853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:43923 every 8 connection(s)
I20260923 14:37:08.563115 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:37:08.567836 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.564499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:32849 (local address 127.2.218.62:36337)
0923 14:37:08.564615 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:08.564619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:08.565426 (+ 807us) server_negotiation.cc:407] Connection header received
0923 14:37:08.566549 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:08.566551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:08.566584 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:08.566647 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:08.567516 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:08.567518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:08.567520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:37:08.567529 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:08.567544 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:37:08.567551 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:08.567613 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:08.567716 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 14:37:08.567747 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:37:08.568501 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.564791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36337 (local address 127.2.218.1:32849)
0923 14:37:08.565306 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:08.565336 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:08.566373 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:08.566778 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:08.566783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:08.566852 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:08.567324 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:37:08.567325 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:08.567356 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:37:08.567409 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:08.567616 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:08.567619 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:08.567622 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:37:08.567725 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 14:37:08.567867 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
I20260923 14:37:08.569439 12854 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36337
I20260923 14:37:08.569624 12854 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:08.569932 12854 heartbeater.cc:511] Master 127.2.218.62:36337 requested a full tablet report, sending...
I20260923 14:37:08.570425 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 12726
I20260923 14:37:08.570536 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:37:08.570807 2920 external_mini_cluster.cc:1428] Setting key 0754a5a6e96936efab8c7cec1bb5f431
I20260923 14:37:08.571311 12666 ts_manager.cc:194] Registered new tserver with Master: 64b995d69ced4dcaa0663388456368ba (127.2.218.1:43923)
I20260923 14:37:08.572669 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36337
--builtin_ntp_servers=127.2.218.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 14:37:08.665058 12857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:08.665242 12857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:08.665274 12857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:08.668892 12857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:08.668936 12857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:08.668980 12857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:37:08.673038 12857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:08.673772 12857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:08.674907 12857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:08.682653 12863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:08.682668 12857 server_base.cc:1034] running on GCE node
W20260923 14:37:08.682705 12865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:08.682663 12862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:08.683233 12857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:08.683768 12857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:08.684933 12857 hybrid_clock.cc:648] HybridClock initialized: now 1790174228684908 us; error 41 us; skew 500 ppm
I20260923 14:37:08.686470 12857 webserver.cc:460] Webserver started at http://127.2.218.2:37161/ using document root <none> and password file <none>
I20260923 14:37:08.686982 12857 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:08.687063 12857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:08.687239 12857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:08.688862 12857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "3cb221814c7547e69446ae85851ce4bd"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "327c981e960250a9fa7d19b30a5607c9"
server_key_iv: "23b9a48647342e193f1aaa3da1b74bb5"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.689370 12857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "3cb221814c7547e69446ae85851ce4bd"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "327c981e960250a9fa7d19b30a5607c9"
server_key_iv: "23b9a48647342e193f1aaa3da1b74bb5"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.692703 12857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:37:08.694784 12871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.695902 12857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:37:08.696045 12857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "3cb221814c7547e69446ae85851ce4bd"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "327c981e960250a9fa7d19b30a5607c9"
server_key_iv: "23b9a48647342e193f1aaa3da1b74bb5"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.696151 12857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:08.700354 12857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:08.700840 12857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:08.701010 12857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:08.701454 12857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:08.702260 12857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:08.702323 12857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.702376 12857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:08.702421 12857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.710330 12857 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:44199
I20260923 14:37:08.710384 12984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:44199 every 8 connection(s)
I20260923 14:37:08.711061 12857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:37:08.716156 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.712578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:46535 (local address 127.2.218.62:36337)
0923 14:37:08.712718 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:08.712721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:08.713514 (+ 793us) server_negotiation.cc:407] Connection header received
0923 14:37:08.714774 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:08.714777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:08.714813 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:08.714883 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:08.715797 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:08.715801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:08.715803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:37:08.715814 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:08.715833 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:37:08.715840 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:08.715910 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:08.716023 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 14:37:08.716059 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:37:08.716835 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.712839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36337 (local address 127.2.218.2:46535)
0923 14:37:08.713408 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:08.713440 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:08.714587 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:08.715023 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:08.715028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:08.715100 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:08.715576 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:37:08.715578 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:08.715609 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:37:08.715668 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:08.715914 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:08.715919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:08.715923 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:37:08.716034 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 14:37:08.716210 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260923 14:37:08.717711 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 12857
I20260923 14:37:08.717857 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:37:08.717935 12985 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36337
I20260923 14:37:08.718180 12985 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:08.718251 2920 external_mini_cluster.cc:1428] Setting key 1856b234bc287a83d0573399207c2de3
I20260923 14:37:08.718588 12985 heartbeater.cc:511] Master 127.2.218.62:36337 requested a full tablet report, sending...
I20260923 14:37:08.719537 12665 ts_manager.cc:194] Registered new tserver with Master: 3cb221814c7547e69446ae85851ce4bd (127.2.218.2:44199)
I20260923 14:37:08.720310 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36337
--builtin_ntp_servers=127.2.218.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 14:37:08.814358 12988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:08.814545 12988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:08.814604 12988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:08.817971 12988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:08.818015 12988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:08.818099 12988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:37:08.821933 12988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:08.822677 12988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:08.823705 12988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:08.831542 12994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:08.831550 12996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:08.831643 12993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:08.831843 12988 server_base.cc:1034] running on GCE node
I20260923 14:37:08.832196 12988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:08.832716 12988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:08.833928 12988 hybrid_clock.cc:648] HybridClock initialized: now 1790174228833903 us; error 42 us; skew 500 ppm
I20260923 14:37:08.835582 12988 webserver.cc:460] Webserver started at http://127.2.218.3:39673/ using document root <none> and password file <none>
I20260923 14:37:08.836056 12988 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:08.836143 12988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:08.836354 12988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:08.838022 12988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "6bd96f73474f4dad888a59bbc89c5dfa"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "46d73cdeacb8044262aa1030e80be5a2"
server_key_iv: "074b2f1dda2353c646eb1138e874093d"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.838621 12988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "6bd96f73474f4dad888a59bbc89c5dfa"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "46d73cdeacb8044262aa1030e80be5a2"
server_key_iv: "074b2f1dda2353c646eb1138e874093d"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.841703 12988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:37:08.843776 13002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.844727 12988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:37:08.844853 12988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "6bd96f73474f4dad888a59bbc89c5dfa"
format_stamp: "Formatted at 2026-09-23 14:37:08 on dist-test-slave-zl47"
server_key: "46d73cdeacb8044262aa1030e80be5a2"
server_key_iv: "074b2f1dda2353c646eb1138e874093d"
server_key_version: "encryptionkey@0"
I20260923 14:37:08.844951 12988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:08.848599 12988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:08.849108 12988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:08.849274 12988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:08.849717 12988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:08.850535 12988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:08.850605 12988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.850666 12988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:08.850701 12988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:08.858937 12988 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:33241
I20260923 14:37:08.858959 13115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:33241 every 8 connection(s)
I20260923 14:37:08.859615 12988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:37:08.864404 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.861085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:45047 (local address 127.2.218.62:36337)
0923 14:37:08.861259 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:08.861264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:08.862013 (+ 749us) server_negotiation.cc:407] Connection header received
0923 14:37:08.863045 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:08.863048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:08.863083 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:08.863168 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:08.864033 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:08.864036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:08.864037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:37:08.864049 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:08.864069 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:37:08.864078 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:08.864166 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:08.864274 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 14:37:08.864308 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:37:08.865084 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 12988
I20260923 14:37:08.865187 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:37:08.865162 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:08.861391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36337 (local address 127.2.218.3:45047)
0923 14:37:08.861897 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:08.861927 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:08.862883 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:08.863289 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:08.863294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:08.863364 (+ 70us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:08.863811 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:37:08.863812 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:08.863841 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:37:08.863891 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:08.864151 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:08.864156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:08.864160 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:37:08.864282 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 14:37:08.864426 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
I20260923 14:37:08.865459 2920 external_mini_cluster.cc:1428] Setting key 6cfd16f486922e6848803a1ac221cf88
I20260923 14:37:08.866263 13116 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36337
I20260923 14:37:08.866461 13116 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:08.866793 13116 heartbeater.cc:511] Master 127.2.218.62:36337 requested a full tablet report, sending...
I20260923 14:37:08.867687 12665 ts_manager.cc:194] Registered new tserver with Master: 6bd96f73474f4dad888a59bbc89c5dfa (127.2.218.3:33241)
I20260923 14:37:08.878796 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 14:37:08.881243 12705 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:37:08.880486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.2.218.62:36337)
0923 14:37:08.880607 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:08.880612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:08.880748 (+ 136us) server_negotiation.cc:407] Connection header received
0923 14:37:08.880843 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:08.880844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:08.880870 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:08.880941 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:08.881144 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55884: BlockingRecv error: recv got EOF from 127.0.0.1:55884 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:37:08.881529 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 12726
I20260923 14:37:08.886808 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 12857
I20260923 14:37:08.891613 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 12988
I20260923 14:37:08.896154 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 12635
2026-09-23T14:37:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (681 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:37:08 dist-test-slave-zl47 krb5kdc[13126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:37:08 dist-test-slave-zl47 krb5kdc[13126](info): set up 2 sockets
Sep 23 14:37:08 dist-test-slave-zl47 krb5kdc[13126](info): commencing operation
krb5kdc: starting...
W20260923 14:37:09.787272 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.872s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:37:09 dist-test-slave-zl47 krb5kdc[13126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:37:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:37:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:37:09.885084 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33361
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:33361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:37:09.977082 13139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:09.977254 13139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:09.977285 13139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:09.980623 13139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:37:09.980671 13139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:09.980691 13139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:09.980731 13139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:37:09.980757 13139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:37:09.984894 13139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:33361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:09.985597 13139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:09.986394 13139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:09.993100 13147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:09.993121 13145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:09.993175 13139 server_base.cc:1034] running on GCE node
W20260923 14:37:09.993100 13144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:09.993816 13139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:09.994735 13139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:09.995911 13139 hybrid_clock.cc:648] HybridClock initialized: now 1790174229995887 us; error 34 us; skew 500 ppm
Sep 23 14:37:09 dist-test-slave-zl47 krb5kdc[13126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174229, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:37:09.998268 13139 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:37:09.998986 13139 webserver.cc:460] Webserver started at http://127.2.218.62:32789/ using document root <none> and password file <none>
I20260923 14:37:09.999418 13139 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:09.999504 13139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:09.999697 13139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:10.001257 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "54427a596981430daa7825be54893ead"
format_stamp: "Formatted at 2026-09-23 14:37:09 on dist-test-slave-zl47"
server_key: "4d7a4d632ce2e082433546740c314049"
server_key_iv: "ee8e1065e269c27512a0363a82eddf90"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.001742 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "54427a596981430daa7825be54893ead"
format_stamp: "Formatted at 2026-09-23 14:37:09 on dist-test-slave-zl47"
server_key: "4d7a4d632ce2e082433546740c314049"
server_key_iv: "ee8e1065e269c27512a0363a82eddf90"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.004974 13139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:37:10.007015 13154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:10.007985 13139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:37:10.008109 13139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "54427a596981430daa7825be54893ead"
format_stamp: "Formatted at 2026-09-23 14:37:09 on dist-test-slave-zl47"
server_key: "4d7a4d632ce2e082433546740c314049"
server_key_iv: "ee8e1065e269c27512a0363a82eddf90"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.008201 13139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:10.026551 13139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:10.029093 13139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:10.029253 13139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:10.035490 13139 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:33361
I20260923 14:37:10.035709 13206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:33361 every 8 connection(s)
I20260923 14:37:10.036316 13139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:37:10.039144 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:37:10.039247 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 13139
I20260923 14:37:10.039347 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:37:10.039613 2920 external_mini_cluster.cc:1428] Setting key 6750674906c8caa8691f6c5e261b6a63
I20260923 14:37:10.045871 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead: Bootstrap starting.
Sep 23 14:37:10 dist-test-slave-zl47 krb5kdc[13126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:37:10.047739 13207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead: Neither blocks nor log segments found. Creating new log.
I20260923 14:37:10.048553 13207 log.cc:824] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead: Log is configured to *not* fsync() on all Append() calls
I20260923 14:37:10.050371 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead: No bootstrap required, opened a new log
I20260923 14:37:10.052403 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.040769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.2.218.62:33361)
0923 14:37:10.041129 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.041143 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.041176 (+ 33us) server_negotiation.cc:407] Connection header received
0923 14:37:10.041919 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.041941 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.042295 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.042591 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:10.043147 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.044389 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.045090 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.045637 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.045780 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.047846 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:10.047852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:10.047867 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:37:10.047893 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:10.049483 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:10.049903 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:10.049908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:10.049913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:10.050023 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:10.050410 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:10.050415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:10.050418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:10.050852 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:10.051029 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.051330 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.051599 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":128,"threads_started":1}
I20260923 14:37:10.054404 13207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54427a596981430daa7825be54893ead" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33361 } }
I20260923 14:37:10.054612 13207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:37:10.054653 13207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54427a596981430daa7825be54893ead, State: Initialized, Role: FOLLOWER
I20260923 14:37:10.054977 13207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54427a596981430daa7825be54893ead" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33361 } }
I20260923 14:37:10.055075 13207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:37:10.055112 13207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:37:10.055169 13207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:37:10.055826 13207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54427a596981430daa7825be54893ead" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33361 } }
I20260923 14:37:10.056061 13207 leader_election.cc:302] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54427a596981430daa7825be54893ead; no voters:
I20260923 14:37:10.056264 13207 leader_election.cc:288] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:37:10.056399 13212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:37:10.056681 13212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [term 1 LEADER]: Becoming Leader. State: Replica: 54427a596981430daa7825be54893ead, State: Running, Role: LEADER
I20260923 14:37:10.056939 13212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54427a596981430daa7825be54893ead" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33361 } }
I20260923 14:37:10.057314 13207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:37:10.058323 13214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54427a596981430daa7825be54893ead. Latest consensus state: current_term: 1 leader_uuid: "54427a596981430daa7825be54893ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54427a596981430daa7825be54893ead" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33361 } } }
I20260923 14:37:10.058682 13214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [sys.catalog]: This master's current role is: LEADER
I20260923 14:37:10.058571 13213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54427a596981430daa7825be54893ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54427a596981430daa7825be54893ead" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33361 } } }
I20260923 14:37:10.059002 13213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead [sys.catalog]: This master's current role is: LEADER
I20260923 14:37:10.059562 13226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:37:10.061623 13226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:37:10.066673 13226 catalog_manager.cc:1347] Generated new cluster ID: a7ec8148267f4ed4af45100865975a2c
I20260923 14:37:10.066730 13226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:37:10.075481 13226 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:37:10.076439 13226 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:37:10.081276 13226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead: Generated new TSK 0
I20260923 14:37:10.081821 13226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:37:10.104410 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33361
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:37:10.197808 13233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:10.198041 13233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:10.198079 13233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:10.201423 13233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:10.201462 13233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:10.201542 13233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:37:10.205677 13233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:10.206476 13233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:10.207234 13233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:10.215039 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:10.215076 13233 server_base.cc:1034] running on GCE node
W20260923 14:37:10.215106 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:10.215127 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:10.215632 13233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:10.216075 13233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:10.217218 13233 hybrid_clock.cc:648] HybridClock initialized: now 1790174230217202 us; error 25 us; skew 500 ppm
Sep 23 14:37:10 dist-test-slave-zl47 krb5kdc[13126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:37:10.219440 13233 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:37:10.220119 13233 webserver.cc:460] Webserver started at http://127.2.218.1:45295/ using document root <none> and password file <none>
I20260923 14:37:10.220575 13233 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:10.220636 13233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:10.220819 13233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:10.222515 13233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "c90c71039c374f769aa049ca2c5c3d55"
format_stamp: "Formatted at 2026-09-23 14:37:10 on dist-test-slave-zl47"
server_key: "76f48f7790d5372c5f349349766232ba"
server_key_iv: "eb34d14606acbb36362586f48b59c37b"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.222975 13233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "c90c71039c374f769aa049ca2c5c3d55"
format_stamp: "Formatted at 2026-09-23 14:37:10 on dist-test-slave-zl47"
server_key: "76f48f7790d5372c5f349349766232ba"
server_key_iv: "eb34d14606acbb36362586f48b59c37b"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.225924 13233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:37:10.227818 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:10.228858 13233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:37:10.228935 13233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "c90c71039c374f769aa049ca2c5c3d55"
format_stamp: "Formatted at 2026-09-23 14:37:10 on dist-test-slave-zl47"
server_key: "76f48f7790d5372c5f349349766232ba"
server_key_iv: "eb34d14606acbb36362586f48b59c37b"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.228999 13233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:10.236855 13233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:10.239132 13233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:10.239246 13233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:10.239624 13233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:10.240211 13233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:10.240249 13233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:10.240279 13233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:10.240291 13233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:10.247355 13233 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:34849
I20260923 14:37:10.247413 13361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:34849 every 8 connection(s)
I20260923 14:37:10.248031 13233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:37:10.248670 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 13233
I20260923 14:37:10.248816 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:37:10.249055 2920 external_mini_cluster.cc:1428] Setting key 5cdea55dbaff1d06751eb9635c481890
Sep 23 14:37:10 dist-test-slave-zl47 krb5kdc[13126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
I20260923 14:37:10.261238 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.249534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:54461 (local address 127.2.218.62:33361)
0923 14:37:10.249662 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.249666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.250441 (+ 775us) server_negotiation.cc:407] Connection header received
0923 14:37:10.252103 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.252108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.252166 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.252216 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:10.253241 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.253742 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.255380 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.255634 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.255738 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.258338 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:10.258341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:10.258344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:37:10.258368 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:10.259586 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:10.260017 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:10.260020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:10.260021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:10.260082 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:10.260392 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:10.260394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:10.260395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:10.260533 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:10.260723 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.261035 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.261137 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:37:10.262051 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.249853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33361 (local address 127.2.218.1:54461)
0923 14:37:10.250309 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:10.250364 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:10.251886 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:10.252491 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:10.252496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:10.252769 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:10.253115 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.253125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.253877 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.253883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.255260 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.255272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.255806 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.255813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.256044 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.256600 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:10.256623 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:10.258201 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:10.259694 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:10.259698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:10.259706 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:10.259911 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:10.260165 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:10.260167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:10.260169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:10.260313 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:10.260608 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:10.260612 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:10.260785 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 14:37:10.260953 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 14:37:10.261224 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260923 14:37:10.263057 13362 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33361
I20260923 14:37:10.263247 13362 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:10.263602 13362 heartbeater.cc:511] Master 127.2.218.62:33361 requested a full tablet report, sending...
I20260923 14:37:10.264804 13171 ts_manager.cc:194] Registered new tserver with Master: c90c71039c374f769aa049ca2c5c3d55 (127.2.218.1:34849)
I20260923 14:37:10.265725 13171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:54461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:37:10.273775 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33361
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:37:10.365816 13367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:10.366019 13367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:10.366076 13367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:10.369349 13367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:10.369414 13367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:10.369483 13367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:37:10.373786 13367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:10.374624 13367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:10.375379 13367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:10.382725 13372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:10.382723 13375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:10.382822 13367 server_base.cc:1034] running on GCE node
W20260923 14:37:10.382743 13373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:10.383239 13367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:10.383705 13367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:10.384939 13367 hybrid_clock.cc:648] HybridClock initialized: now 1790174230384913 us; error 28 us; skew 500 ppm
Sep 23 14:37:10 dist-test-slave-zl47 krb5kdc[13126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:37:10.387362 13367 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:37:10.387959 13367 webserver.cc:460] Webserver started at http://127.2.218.2:32855/ using document root <none> and password file <none>
I20260923 14:37:10.388337 13367 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:10.388384 13367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:10.388505 13367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:10.389956 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1"
format_stamp: "Formatted at 2026-09-23 14:37:10 on dist-test-slave-zl47"
server_key: "5150271d967890f818be7fe051e53a07"
server_key_iv: "26424f2ba8070a589a0b2e4b6c35c660"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.390455 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1"
format_stamp: "Formatted at 2026-09-23 14:37:10 on dist-test-slave-zl47"
server_key: "5150271d967890f818be7fe051e53a07"
server_key_iv: "26424f2ba8070a589a0b2e4b6c35c660"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.393206 13367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:37:10.395073 13382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:10.395973 13367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:37:10.396067 13367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1"
format_stamp: "Formatted at 2026-09-23 14:37:10 on dist-test-slave-zl47"
server_key: "5150271d967890f818be7fe051e53a07"
server_key_iv: "26424f2ba8070a589a0b2e4b6c35c660"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.396148 13367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:10.426524 13367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:10.428716 13367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:10.428866 13367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:10.429325 13367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:10.429958 13367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:10.429996 13367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:10.430047 13367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:10.430073 13367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:10.437647 13367 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:40031
I20260923 14:37:10.437705 13495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:40031 every 8 connection(s)
I20260923 14:37:10.438324 13367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
Sep 23 14:37:10 dist-test-slave-zl47 krb5kdc[13126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:37:10.448477 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 13367
I20260923 14:37:10.448581 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:37:10.448889 2920 external_mini_cluster.cc:1428] Setting key 7b7a0d37bc52bad2329455ca7bcf102d
I20260923 14:37:10.449060 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.439694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:34485 (local address 127.2.218.62:33361)
0923 14:37:10.439801 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.439804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.440455 (+ 651us) server_negotiation.cc:407] Connection header received
0923 14:37:10.441474 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.441477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.441517 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.441578 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:10.442593 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.443031 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.443997 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.444206 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.444277 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.446219 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:10.446222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:10.446225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:37:10.446247 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:10.447349 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:10.447731 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:10.447733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:10.447734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:10.447804 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:10.448045 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:10.448046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:10.448047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:10.448231 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:10.448304 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.448797 (+ 493us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.448885 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:37:10.449750 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.439960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33361 (local address 127.2.218.2:34485)
0923 14:37:10.440333 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:10.440365 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:10.441319 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:10.441675 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:10.441680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:10.441945 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:10.442470 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.442485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.443121 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.443123 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.443897 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.443905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.444313 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.444315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.444427 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.444883 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:10.444896 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:10.446013 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:10.447443 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:10.447447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:10.447455 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:10.447650 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:10.447874 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:10.447876 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:10.447877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:10.447977 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:10.448322 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:10.448326 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:10.448559 (+ 233us) client_negotiation.cc:769] Sending connection context
0923 14:37:10.448737 (+ 178us) client_negotiation.cc:240] Negotiation successful
0923 14:37:10.448996 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":125,"threads_started":1}
I20260923 14:37:10.450774 13496 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33361
I20260923 14:37:10.450964 13496 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:10.451315 13496 heartbeater.cc:511] Master 127.2.218.62:33361 requested a full tablet report, sending...
I20260923 14:37:10.452101 13171 ts_manager.cc:194] Registered new tserver with Master: 189c6ffe3d1f4a299390a24ef9a1b5d1 (127.2.218.2:40031)
I20260923 14:37:10.452560 13171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:34485
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:37:10.470315 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33361
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:45565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:37:10.563230 13501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:10.563434 13501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:10.563489 13501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:10.566817 13501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:10.566875 13501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:10.566946 13501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:37:10.571105 13501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:10.571839 13501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:10.572585 13501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:10.580323 13506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:10.580400 13509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:10.580552 13501 server_base.cc:1034] running on GCE node
W20260923 14:37:10.580608 13507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:10.580945 13501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:10.581465 13501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:10.582643 13501 hybrid_clock.cc:648] HybridClock initialized: now 1790174230582621 us; error 38 us; skew 500 ppm
Sep 23 14:37:10 dist-test-slave-zl47 krb5kdc[13126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:37:10.585006 13501 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:37:10.585649 13501 webserver.cc:460] Webserver started at http://127.2.218.3:41987/ using document root <none> and password file <none>
I20260923 14:37:10.586050 13501 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:10.586161 13501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:10.586328 13501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:37:10.587942 13501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "ee455a9d26ac40598210e20e1f26984c"
format_stamp: "Formatted at 2026-09-23 14:37:10 on dist-test-slave-zl47"
server_key: "798977e3c763d6aaa11669700bccafac"
server_key_iv: "cddef7d66751ec33d9c8e2761cd107fc"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.588431 13501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "ee455a9d26ac40598210e20e1f26984c"
format_stamp: "Formatted at 2026-09-23 14:37:10 on dist-test-slave-zl47"
server_key: "798977e3c763d6aaa11669700bccafac"
server_key_iv: "cddef7d66751ec33d9c8e2761cd107fc"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.591429 13501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:37:10.593321 13516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:10.594399 13501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:37:10.594502 13501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "ee455a9d26ac40598210e20e1f26984c"
format_stamp: "Formatted at 2026-09-23 14:37:10 on dist-test-slave-zl47"
server_key: "798977e3c763d6aaa11669700bccafac"
server_key_iv: "cddef7d66751ec33d9c8e2761cd107fc"
server_key_version: "encryptionkey@0"
I20260923 14:37:10.594589 13501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:10.605273 13501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:10.607642 13501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:10.607800 13501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:10.608304 13501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:10.608963 13501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:10.609001 13501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:10.609056 13501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:10.609090 13501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:10.616626 13501 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:43141
I20260923 14:37:10.616652 13629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:43141 every 8 connection(s)
I20260923 14:37:10.617280 13501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
Sep 23 14:37:10 dist-test-slave-zl47 krb5kdc[13126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:37:10.624785 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 13501
I20260923 14:37:10.624886 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:37:10.625155 2920 external_mini_cluster.cc:1428] Setting key 53a35dc9ed49fc808b3c435a21e68586
I20260923 14:37:10.628057 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.618631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:42781 (local address 127.2.218.62:33361)
0923 14:37:10.618742 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.618746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.619365 (+ 619us) server_negotiation.cc:407] Connection header received
0923 14:37:10.620367 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.620369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.620410 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.620471 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:10.621358 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.621802 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.622760 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.622938 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.623006 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.625073 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:10.625075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:10.625077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:37:10.625097 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:10.626549 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:10.626975 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:10.626978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:10.626979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:10.627057 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:10.627349 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:10.627350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:10.627351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:10.627486 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:10.627550 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.627889 (+ 339us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.627962 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:37:10.628674 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.618900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33361 (local address 127.2.218.3:42781)
0923 14:37:10.619246 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:10.619278 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:10.620212 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:10.620605 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:10.620610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:10.620874 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:10.621251 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.621262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.621890 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.621892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.622671 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.622679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.623065 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.623068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.623218 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.623590 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:10.623602 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:10.624883 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:10.626642 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:10.626646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:10.626654 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:10.626885 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:10.627143 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:10.627146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:10.627148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:10.627276 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:10.627560 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:10.627564 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:10.627727 (+ 163us) client_negotiation.cc:769] Sending connection context
0923 14:37:10.627848 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 14:37:10.628025 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260923 14:37:10.629513 13630 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33361
I20260923 14:37:10.629671 13630 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:10.630002 13630 heartbeater.cc:511] Master 127.2.218.62:33361 requested a full tablet report, sending...
I20260923 14:37:10.630851 13171 ts_manager.cc:194] Registered new tserver with Master: ee455a9d26ac40598210e20e1f26984c (127.2.218.3:43141)
I20260923 14:37:10.631283 13171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:42781
I20260923 14:37:10.638422 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:37:10.646708 13210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.640197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.2.218.62:33361)
0923 14:37:10.640321 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.640325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.640504 (+ 179us) server_negotiation.cc:407] Connection header received
0923 14:37:10.640604 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.640606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.640635 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.640690 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:10.641306 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.641721 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.642542 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.642724 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.642791 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.643941 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:10.643943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:10.643945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:37:10.643960 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:10.645130 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:10.645495 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:10.645497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:10.645498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:10.645565 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:10.645768 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:10.645770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:10.645770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:10.645937 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:10.646031 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.646382 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.646523 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:37:10.649075 13171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:37:10.651757 13171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:37:10.663897 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.658821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:43141 (local address 127.0.0.1:49280)
0923 14:37:10.659366 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:10.659378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:10.659461 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:10.660044 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:10.660049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:10.660068 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:37:10.660177 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.660185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.661103 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.661106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.662450 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.662461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.663546 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.663549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.663671 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.663687 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 14:37:10.663738 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 14:37:10.663787 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":403,"thread_start_us":91,"threads_started":1}
I20260923 14:37:10.663897 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.658671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:34849 (local address 127.0.0.1:42004)
0923 14:37:10.658980 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:10.659020 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:10.659142 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:10.660240 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:10.660246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:10.660266 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:37:10.660341 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.660346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.661532 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.661537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.662462 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.662472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.663432 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.663436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.663542 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.663574 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 14:37:10.663663 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 14:37:10.663771 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":22,"thread_start_us":67,"threads_started":1}
I20260923 14:37:10.664227 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.658970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.2.218.3:43141)
0923 14:37:10.659262 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.659273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.659440 (+ 167us) server_negotiation.cc:407] Connection header received
0923 14:37:10.659581 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.659586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.659758 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.659889 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:37:10.660293 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.660980 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.662774 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.663292 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.663347 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.663466 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.664066 (+ 600us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.664136 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":111,"threads_started":1}
I20260923 14:37:10.663920 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.658771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.2.218.1:34849)
0923 14:37:10.659844 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.659854 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.659874 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:37:10.659940 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.659946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.660125 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.660266 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:37:10.660436 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.661422 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.662601 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.663317 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.663410 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.663513 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.663707 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.663801 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":116,"threads_started":1}
I20260923 14:37:10.665112 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.659759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40031 (local address 127.0.0.1:58882)
0923 14:37:10.660437 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:10.660448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:10.660520 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:10.661262 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:10.661265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:10.661279 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:37:10.661356 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.661361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.662712 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.662714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.663700 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.663708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.664771 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.664773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.664854 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.664876 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:37:10.664953 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 14:37:10.665007 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":46,"thread_start_us":70,"threads_started":1}
I20260923 14:37:10.665275 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.660110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.2.218.2:40031)
0923 14:37:10.660727 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.660737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.660756 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:37:10.660818 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.660825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.660993 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.661116 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:37:10.661494 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.662583 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.663815 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.664664 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.664765 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.664873 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.664978 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.665054 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":97,"threads_started":1}
I20260923 14:37:10.666761 13296 tablet_service.cc:1467] Processing CreateTablet for tablet a182f65955604deb8833cee6e5417121 (DEFAULT_TABLE table=test-table [id=d3248dcd895f42a48c242f67717d1f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:37:10.666839 13564 tablet_service.cc:1467] Processing CreateTablet for tablet a182f65955604deb8833cee6e5417121 (DEFAULT_TABLE table=test-table [id=d3248dcd895f42a48c242f67717d1f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:37:10.667199 13430 tablet_service.cc:1467] Processing CreateTablet for tablet a182f65955604deb8833cee6e5417121 (DEFAULT_TABLE table=test-table [id=d3248dcd895f42a48c242f67717d1f7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:37:10.667759 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a182f65955604deb8833cee6e5417121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:37:10.667757 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a182f65955604deb8833cee6e5417121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:37:10.668076 13430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a182f65955604deb8833cee6e5417121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:37:10.673182 13647 tablet_bootstrap.cc:492] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: Bootstrap starting.
I20260923 14:37:10.674250 13649 tablet_bootstrap.cc:492] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: Bootstrap starting.
I20260923 14:37:10.675935 13649 tablet_bootstrap.cc:654] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: Neither blocks nor log segments found. Creating new log.
I20260923 14:37:10.676147 13647 tablet_bootstrap.cc:654] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: Neither blocks nor log segments found. Creating new log.
I20260923 14:37:10.676940 13647 log.cc:824] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: Log is configured to *not* fsync() on all Append() calls
I20260923 14:37:10.676940 13649 log.cc:824] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: Log is configured to *not* fsync() on all Append() calls
I20260923 14:37:10.677469 13648 tablet_bootstrap.cc:492] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55: Bootstrap starting.
I20260923 14:37:10.678520 13649 tablet_bootstrap.cc:492] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: No bootstrap required, opened a new log
I20260923 14:37:10.678675 13647 tablet_bootstrap.cc:492] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: No bootstrap required, opened a new log
I20260923 14:37:10.678691 13649 ts_tablet_manager.cc:1397] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260923 14:37:10.678839 13647 ts_tablet_manager.cc:1397] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:37:10.679621 13648 tablet_bootstrap.cc:654] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55: Neither blocks nor log segments found. Creating new log.
I20260923 14:37:10.680395 13648 log.cc:824] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55: Log is configured to *not* fsync() on all Append() calls
I20260923 14:37:10.681691 13649 raft_consensus.cc:348] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 } } peers { permanent_uuid: "ee455a9d26ac40598210e20e1f26984c" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43141 } } peers { permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 } }
I20260923 14:37:10.681885 13649 raft_consensus.cc:374] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:37:10.681936 13649 raft_consensus.cc:729] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee455a9d26ac40598210e20e1f26984c, State: Initialized, Role: FOLLOWER
I20260923 14:37:10.682022 13648 tablet_bootstrap.cc:492] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55: No bootstrap required, opened a new log
I20260923 14:37:10.682214 13648 ts_tablet_manager.cc:1397] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:37:10.682196 13647 raft_consensus.cc:348] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 } } peers { permanent_uuid: "ee455a9d26ac40598210e20e1f26984c" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43141 } } peers { permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 } }
I20260923 14:37:10.682415 13647 raft_consensus.cc:374] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:37:10.682466 13647 raft_consensus.cc:729] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 189c6ffe3d1f4a299390a24ef9a1b5d1, State: Initialized, Role: FOLLOWER
I20260923 14:37:10.682387 13649 consensus_queue.cc:260] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 } } peers { permanent_uuid: "ee455a9d26ac40598210e20e1f26984c" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43141 } } peers { permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 } }
I20260923 14:37:10.682813 13647 consensus_queue.cc:260] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 } } peers { permanent_uuid: "ee455a9d26ac40598210e20e1f26984c" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43141 } } peers { permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 } }
I20260923 14:37:10.683202 13649 ts_tablet_manager.cc:1428] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260923 14:37:10.683389 13496 heartbeater.cc:503] Master 127.2.218.62:33361 was elected leader, sending a full tablet report...
I20260923 14:37:10.683450 13630 heartbeater.cc:503] Master 127.2.218.62:33361 was elected leader, sending a full tablet report...
I20260923 14:37:10.684381 13647 ts_tablet_manager.cc:1428] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:37:10.685292 13648 raft_consensus.cc:348] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 } } peers { permanent_uuid: "ee455a9d26ac40598210e20e1f26984c" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43141 } } peers { permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 } }
I20260923 14:37:10.685467 13648 raft_consensus.cc:374] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:37:10.685518 13648 raft_consensus.cc:729] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c90c71039c374f769aa049ca2c5c3d55, State: Initialized, Role: FOLLOWER
I20260923 14:37:10.685941 13648 consensus_queue.cc:260] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 } } peers { permanent_uuid: "ee455a9d26ac40598210e20e1f26984c" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43141 } } peers { permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 } }
I20260923 14:37:10.686448 13362 heartbeater.cc:503] Master 127.2.218.62:33361 was elected leader, sending a full tablet report...
I20260923 14:37:10.686705 13648 ts_tablet_manager.cc:1428] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260923 14:37:10.689016 13497 tablet.cc:2369] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:37:10.747737 13653 raft_consensus.cc:482] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:37:10.747869 13653 raft_consensus.cc:504] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 } } peers { permanent_uuid: "ee455a9d26ac40598210e20e1f26984c" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43141 } } peers { permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 } }
I20260923 14:37:10.748651 13653 leader_election.cc:288] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c90c71039c374f769aa049ca2c5c3d55 (127.2.218.1:34849), 189c6ffe3d1f4a299390a24ef9a1b5d1 (127.2.218.2:40031)
W20260923 14:37:10.748986 13363 tablet.cc:2369] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:37:10.752130 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.748928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:51623 (local address 127.2.218.1:34849)
0923 14:37:10.749033 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.749037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.749048 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:37:10.749160 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.749162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.749207 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.749270 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:37:10.749623 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.750234 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.751201 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.751667 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.751752 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.751803 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.751977 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.752038 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:37:10.752130 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.748841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:34849 (local address 127.2.218.3:51623)
0923 14:37:10.748962 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:10.748977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:10.749052 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:10.749373 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:10.749376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:10.749393 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:37:10.749500 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.749504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.750340 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.750343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.751071 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.751078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.751765 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.751768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.751863 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.751883 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:37:10.751961 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:37:10.752020 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":4}
I20260923 14:37:10.752326 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.749289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:37561 (local address 127.2.218.2:40031)
0923 14:37:10.749383 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.749387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.749431 (+ 44us) server_negotiation.cc:407] Connection header received
0923 14:37:10.749592 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.749595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.749643 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.749692 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:37:10.749936 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.750399 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.751328 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.751863 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.751961 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.752012 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.752185 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.752242 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:37:10.752336 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.749067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40031 (local address 127.2.218.3:37561)
0923 14:37:10.749364 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:10.749376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:10.749479 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:10.749729 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:10.749732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:10.749747 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:37:10.749847 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.749852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.750488 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.750490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.751163 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:10.751172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.751973 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.751976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:10.752072 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.752087 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 14:37:10.752159 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 14:37:10.752224 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":85,"threads_started":1}
I20260923 14:37:10.752673 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a182f65955604deb8833cee6e5417121" candidate_uuid: "ee455a9d26ac40598210e20e1f26984c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c90c71039c374f769aa049ca2c5c3d55" is_pre_election: true
I20260923 14:37:10.752866 13450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a182f65955604deb8833cee6e5417121" candidate_uuid: "ee455a9d26ac40598210e20e1f26984c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" is_pre_election: true
I20260923 14:37:10.752928 13316 raft_consensus.cc:2454] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee455a9d26ac40598210e20e1f26984c in term 0.
I20260923 14:37:10.753096 13450 raft_consensus.cc:2454] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee455a9d26ac40598210e20e1f26984c in term 0.
I20260923 14:37:10.753316 13520 leader_election.cc:302] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c90c71039c374f769aa049ca2c5c3d55, ee455a9d26ac40598210e20e1f26984c; no voters:
I20260923 14:37:10.753538 13653 raft_consensus.cc:2781] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:37:10.753615 13653 raft_consensus.cc:482] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:37:10.753667 13653 raft_consensus.cc:3037] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:37:10.754726 13653 raft_consensus.cc:504] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 } } peers { permanent_uuid: "ee455a9d26ac40598210e20e1f26984c" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43141 } } peers { permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 } }
I20260923 14:37:10.755033 13653 leader_election.cc:288] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [CANDIDATE]: Term 1 election: Requested vote from peers c90c71039c374f769aa049ca2c5c3d55 (127.2.218.1:34849), 189c6ffe3d1f4a299390a24ef9a1b5d1 (127.2.218.2:40031)
I20260923 14:37:10.755316 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a182f65955604deb8833cee6e5417121" candidate_uuid: "ee455a9d26ac40598210e20e1f26984c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c90c71039c374f769aa049ca2c5c3d55"
I20260923 14:37:10.755385 13450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a182f65955604deb8833cee6e5417121" candidate_uuid: "ee455a9d26ac40598210e20e1f26984c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1"
I20260923 14:37:10.755417 13316 raft_consensus.cc:3037] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:37:10.755481 13450 raft_consensus.cc:3037] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:37:10.756495 13316 raft_consensus.cc:2454] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee455a9d26ac40598210e20e1f26984c in term 1.
I20260923 14:37:10.756495 13450 raft_consensus.cc:2454] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee455a9d26ac40598210e20e1f26984c in term 1.
I20260923 14:37:10.756742 13520 leader_election.cc:302] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 189c6ffe3d1f4a299390a24ef9a1b5d1, ee455a9d26ac40598210e20e1f26984c; no voters:
I20260923 14:37:10.756871 13653 raft_consensus.cc:2781] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:37:10.757050 13653 raft_consensus.cc:686] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 1 LEADER]: Becoming Leader. State: Replica: ee455a9d26ac40598210e20e1f26984c, State: Running, Role: LEADER
I20260923 14:37:10.757331 13653 consensus_queue.cc:237] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 } } peers { permanent_uuid: "ee455a9d26ac40598210e20e1f26984c" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43141 } } peers { permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 } }
I20260923 14:37:10.760221 13169 catalog_manager.cc:5495] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c reported cstate change: term changed from 0 to 1, leader changed from <none> to ee455a9d26ac40598210e20e1f26984c (127.2.218.3). New cstate: current_term: 1 leader_uuid: "ee455a9d26ac40598210e20e1f26984c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee455a9d26ac40598210e20e1f26984c" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 } health_report { overall_health: UNKNOWN } } }
I20260923 14:37:10.782265 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.778874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.2.218.3:43141)
0923 14:37:10.779079 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:10.779083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:10.779122 (+ 39us) server_negotiation.cc:407] Connection header received
0923 14:37:10.779309 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:10.779311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:10.779352 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:10.779439 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:37:10.779659 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.780139 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.780975 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:10.781168 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:10.781300 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:10.781574 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:37:10.781902 (+ 328us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:37:10.782027 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:10.782083 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 14:37:10.782144 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260923 14:37:10.786978 13316 raft_consensus.cc:1261] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55 [term 1 FOLLOWER]: Refusing update from remote peer ee455a9d26ac40598210e20e1f26984c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:37:10.786978 13450 raft_consensus.cc:1261] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1 [term 1 FOLLOWER]: Refusing update from remote peer ee455a9d26ac40598210e20e1f26984c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:37:10.787587 13657 consensus_queue.cc:1035] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c90c71039c374f769aa049ca2c5c3d55" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 34849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:37:10.787703 13653 consensus_queue.cc:1035] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [LEADER]: Connected to new peer: Peer: permanent_uuid: "189c6ffe3d1f4a299390a24ef9a1b5d1" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:37:10.796588 13661 mvcc.cc:205] Tried to move back new op lower bound from 7332553649297342464 to 7332553649184694272. Current Snapshot: MvccSnapshot[applied={T|T < 7332553649297342464}]
I20260923 14:37:10.800999 13171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:37:10.801141 13171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:37:10.803200 13171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54427a596981430daa7825be54893ead: Sending DeleteTablet for 3 replicas of tablet a182f65955604deb8833cee6e5417121
I20260923 14:37:10.803838 13564 tablet_service.cc:1514] Processing DeleteTablet for tablet a182f65955604deb8833cee6e5417121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:37:10 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:49280
I20260923 14:37:10.803889 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet a182f65955604deb8833cee6e5417121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:37:10 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:42004
I20260923 14:37:10.803989 13430 tablet_service.cc:1514] Processing DeleteTablet for tablet a182f65955604deb8833cee6e5417121 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:37:10 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:58882
I20260923 14:37:10.804312 13671 tablet_replica.cc:330] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55: stopping tablet replica
I20260923 14:37:10.804358 13672 tablet_replica.cc:330] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: stopping tablet replica
I20260923 14:37:10.804610 13672 raft_consensus.cc:2229] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:37:10.804618 13671 raft_consensus.cc:2229] T a182f65955604deb8833cee6e5417121 P c90c71039c374f769aa049ca2c5c3d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:37:10.804840 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 13233
I20260923 14:37:10.804863 13672 raft_consensus.cc:2258] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:37:10.806665 13670 tablet_replica.cc:330] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: stopping tablet replica
I20260923 14:37:10.806965 13670 raft_consensus.cc:2229] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:37:10.807288 13670 raft_consensus.cc:2258] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:37:10.808218 13672 ts_tablet_manager.cc:1905] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:37:10.808481 13670 ts_tablet_manager.cc:1905] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:37:10.810721 13672 ts_tablet_manager.cc:1918] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:37:10.810866 13672 log.cc:1196] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/a182f65955604deb8833cee6e5417121
I20260923 14:37:10.811225 13672 ts_tablet_manager.cc:1939] T a182f65955604deb8833cee6e5417121 P 189c6ffe3d1f4a299390a24ef9a1b5d1: Deleting consensus metadata
I20260923 14:37:10.811360 13670 ts_tablet_manager.cc:1918] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:37:10.811451 13670 log.cc:1196] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/a182f65955604deb8833cee6e5417121
I20260923 14:37:10.811779 13670 ts_tablet_manager.cc:1939] T a182f65955604deb8833cee6e5417121 P ee455a9d26ac40598210e20e1f26984c: Deleting consensus metadata
W20260923 14:37:10.812343 13158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:34849 (error 108)
I20260923 14:37:10.812678 13155 catalog_manager.cc:4841] TS ee455a9d26ac40598210e20e1f26984c (127.2.218.3:43141): tablet a182f65955604deb8833cee6e5417121 (table test-table [id=d3248dcd895f42a48c242f67717d1f7e]) successfully deleted
I20260923 14:37:10.812711 13158 catalog_manager.cc:4841] TS 189c6ffe3d1f4a299390a24ef9a1b5d1 (127.2.218.2:40031): tablet a182f65955604deb8833cee6e5417121 (table test-table [id=d3248dcd895f42a48c242f67717d1f7e]) successfully deleted
I20260923 14:37:10.813088 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 13367
I20260923 14:37:10.813301 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:10.813050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:34849 (local address 127.0.0.1:42010)
0923 14:37:10.813148 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:10.813224 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:34849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 14:37:10.813438 13158 catalog_manager.cc:4584] TS c90c71039c374f769aa049ca2c5c3d55 (127.2.218.1:34849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a182f65955604deb8833cee6e5417121: Network error: Client connection negotiation failed: client connection to 127.2.218.1:34849: connect: Connection refused (error 111)
I20260923 14:37:10.818159 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 13501
I20260923 14:37:10.822414 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 13139
2026-09-23T14:37:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1926 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:37:10 dist-test-slave-zl47 krb5kdc[13676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:37:10 dist-test-slave-zl47 krb5kdc[13676](info): set up 2 sockets
Sep 23 14:37:10 dist-test-slave-zl47 krb5kdc[13676](info): commencing operation
krb5kdc: starting...
W20260923 14:37:11.783761 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:37:11 dist-test-slave-zl47 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:37:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:37:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/rangeradmin_127.2.218.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/rangeradmin_127.2.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/rangerlookup_127.2.218.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/rangerlookup_127.2.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/HTTP_127.2.218.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/HTTP_127.2.218.20@KRBTEST.COM.keytab.
I20260923 14:37:11.928496 2920 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3TRGqf/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/postgres -l logfile start
2026-09-23 14:37:14.072 UTC [13706] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 14:37:14.072 UTC [13706] LOG: listening on IPv4 address "127.2.218.20", port 35749
2026-09-23 14:37:14.075 UTC [13706] LOG: listening on Unix socket "/tmp/.s.PGSQL.35749"
2026-09-23 14:37:14.093 UTC [13709] LOG: database system was shut down at 2026-09-23 14:37:12 UTC
2026-09-23 14:37:14.096 UTC [13706] LOG: database system is ready to accept connections
I20260923 14:37:14.943378 2920 mini_postgres.cc:96] Postgres bound to 35749
2026-09-23 14:37:14.949 UTC [13717] FATAL: database "slave" does not exist
127.2.218.20:35749 - accepting connections
I20260923 14:37:14.949699 2920 mini_ranger.cc:161] Starting Ranger...
I20260923 14:37:14.962247 2920 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 14:37:15.351061 2920 mini_ranger.cc:87] Created ranger Postgres database
I20260923 14:37:15.351150 2920 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-admin
2026-09-23 14:37:15,661 [I] DB FLAVOR :POSTGRES
2026-09-23 14:37:15,661 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 14:37:15,661 [I] Checking connection..
2026-09-23 14:37:15,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 14:37:15,900 [I] Checking connection passed.
2026-09-23 14:37:15,900 [I] --------- Verifying version history table ---------
2026-09-23 14:37:15,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:37:16,108 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 14:37:16,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:37:16,326 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 14:37:16,326 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 14:37:16,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 14:37:16.521 UTC [13781] WARNING: there is no transaction in progress
2026-09-23 14:37:16,526 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 14:37:16,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:37:16,736 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 14:37:16,736 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 14:37:16,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 14:37:16,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 14:37:17,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zl47.c.gcp-upstream.internal','N') ;"
2026-09-23 14:37:17,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 14:37:17,603 [I] Table x_portal_user does not exist in database ranger
2026-09-23 14:37:17,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 14:37:17,823 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 14:37:17,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 14:37:18,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 14:37:18,256 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 14:37:18,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 14:37:19.071 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.085 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.099 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.113 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.127 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.143 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.191 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.197 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.210 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.218 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.227 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.239 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.248 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.253 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.259 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19.265 UTC [13889] WARNING: there is no transaction in progress
2026-09-23 14:37:19,811 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 14:37:19,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 14:37:20,049 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 14:37:20,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 14:37:20,275 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 14:37:20,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 14:37:20,470 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 14:37:20,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 14:37:20,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 14:37:20,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zl47.c.gcp-upstream.internal';"
2026-09-23 14:37:21,126 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 14:37:21,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 14:37:21,336 [I] DB_PATCHES have already been applied
I20260923 14:37:21.341536 2920 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-admin:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3TRGqf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 14:37:21.341630 2920 mini_ranger.cc:193] Using host: 127.2.218.20
I20260923 14:37:21.344732 2920 mini_ranger.cc:227] Ranger admin URL: http://127.2.218.20:33449
Sep 23, 2026 2:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 2:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 2:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 2:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 2:37:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 2:37:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 2:37:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 14:37:23.269089 2920 mini_ranger.cc:160] Time spent starting Ranger: real 8.319s user 0.001s sys 0.010s
Sep 23, 2026 2:37:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 2:37:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 23, 2026 2:37:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 2:37:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 2:37:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 14:37:41.888844 2920 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/rangerkms_127.2.218.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/rangerkms_127.2.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.218.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.218.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.218.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/HTTP_127.2.218.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.218.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/HTTP_127.2.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 14:37:41 dist-test-slave-zl47 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174261, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 14:37:41.956302 2920 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 14:37:41.964255 2920 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 14:37:42.699851 2920 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 14:37:42.699940 2920 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-kms
I20260923 14:37:42.700033 2920 mini_ranger_kms.cc:227] Using postgres at 127.2.218.20:35749
2026-09-23 14:37:43,082 [I] DB FLAVOR :POSTGRES
2026-09-23 14:37:43,083 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 14:37:43,083 [I] Checking connection
2026-09-23 14:37:43,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 14:37:43,293 [I] connection success
2026-09-23 14:37:43,293 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 14:37:43,293 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 14:37:43,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 14:37:43,531 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 14:37:43,532 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 14:37:43,532 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 14:37:43,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35749/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 14:37:43,756 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 14:37:44.198199 2920 mini_ranger_kms.cc:310] Created kms service
I20260923 14:37:44.372479 2920 mini_ranger_kms.cc:326] Created kudu user
I20260923 14:37:44.407850 2920 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 14:37:44 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174261, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:37:44.787732 2920 mini_ranger_kms.cc:384] Added ranger policy
I20260923 14:37:44.787905 2920 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-kms:/tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 14:37:44.787931 2920 mini_ranger_kms.cc:240] Using host: 127.2.218.20
I20260923 14:37:44.792042 2920 mini_ranger_kms.cc:276] Ranger KMS PID: 14070
I20260923 14:37:44.792145 2920 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.218.20:44951
Sep 23, 2026 2:37:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 2:37:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 2:37:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 2:37:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 2:37:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 2:37:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 14:37:46 dist-test-slave-zl47 krb5kdc[13676](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790174266, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.218.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:37:46.859501 2920 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.903s user 0.011s sys 0.200s
I20260923 14:37:46.859645 2920 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 14:37:46.859754 2920 mini_ranger_kms.cc:401] 127.2.218.20:44951/kms/v1/keys
Sep 23 14:37:47 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174266, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.218.20@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
Sep 23 14:37:47 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174266, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.218.20@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
Sep 23 14:37:49 dist-test-slave-zl47 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:37:49.555047 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33551
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:39493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:33551
--ranger_config_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:44951/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:37:49.653174 14121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:49.653403 14121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:49.653470 14121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:49.656843 14121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:37:49.656919 14121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:49.656955 14121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:49.657009 14121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:37:49.657047 14121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:37:49.661497 14121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:39493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.218.20:44951/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:33551
--ranger_config_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:49.662442 14121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:49.663347 14121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:49.670677 14129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:49.670660 14126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:49.670683 14127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:49.670943 14121 server_base.cc:1034] running on GCE node
I20260923 14:37:49.671489 14121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:49.672438 14121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:49.673620 14121 hybrid_clock.cc:648] HybridClock initialized: now 1790174269673612 us; error 26 us; skew 500 ppm
Sep 23 14:37:49 dist-test-slave-zl47 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174269, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:37:49.680878 14121 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:37:49.681669 14121 webserver.cc:460] Webserver started at http://127.2.218.62:36511/ using document root <none> and password file <none>
I20260923 14:37:49.682216 14121 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:49.682296 14121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:49.682511 14121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:37:49 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174269, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:37:49.732283 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "9ab94dffdb57447c853ebd56df142757"
format_stamp: "Formatted at 2026-09-23 14:37:49 on dist-test-slave-zl47"
server_key: "f112053fc91baaaf8ae830800acc7494"
server_key_iv: "eda5845d3f2ace6cc1c6c7fe577318bc"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:49.733018 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "9ab94dffdb57447c853ebd56df142757"
format_stamp: "Formatted at 2026-09-23 14:37:49 on dist-test-slave-zl47"
server_key: "f112053fc91baaaf8ae830800acc7494"
server_key_iv: "eda5845d3f2ace6cc1c6c7fe577318bc"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:49.737598 14121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 14:37:49.751626 14136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:49.753064 14121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 14:37:49.753178 14121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "9ab94dffdb57447c853ebd56df142757"
format_stamp: "Formatted at 2026-09-23 14:37:49 on dist-test-slave-zl47"
server_key: "f112053fc91baaaf8ae830800acc7494"
server_key_iv: "eda5845d3f2ace6cc1c6c7fe577318bc"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:49.753291 14121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:49.772078 14121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:49.774454 14121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:49.774654 14121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:49.783476 14121 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:33551
I20260923 14:37:49.783497 14188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:33551 every 8 connection(s)
I20260923 14:37:49.784245 14121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:37:49.786691 14189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:37:49.790445 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 14121
I20260923 14:37:49.790536 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:37:49.790819 2920 external_mini_cluster.cc:1428] Setting key db382f15e3318085a0c21aaa20e65ebe
I20260923 14:37:49.791754 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757: Bootstrap starting.
I20260923 14:37:49.793640 14189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757: Neither blocks nor log segments found. Creating new log.
I20260923 14:37:49.794346 14189 log.cc:824] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757: Log is configured to *not* fsync() on all Append() calls
I20260923 14:37:49.795819 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757: No bootstrap required, opened a new log
Sep 23 14:37:49 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:37:49.798952 14189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab94dffdb57447c853ebd56df142757" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33551 } }
I20260923 14:37:49.799201 14189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:37:49.799297 14189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab94dffdb57447c853ebd56df142757, State: Initialized, Role: FOLLOWER
I20260923 14:37:49.799722 14189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab94dffdb57447c853ebd56df142757" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33551 } }
I20260923 14:37:49.799834 14189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:37:49.799921 14189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:37:49.799995 14189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:37:49.800915 14189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab94dffdb57447c853ebd56df142757" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33551 } }
I20260923 14:37:49.801262 14189 leader_election.cc:302] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ab94dffdb57447c853ebd56df142757; no voters:
I20260923 14:37:49.801517 14189 leader_election.cc:288] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:37:49.801586 14194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:37:49.801775 14194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [term 1 LEADER]: Becoming Leader. State: Replica: 9ab94dffdb57447c853ebd56df142757, State: Running, Role: LEADER
I20260923 14:37:49.802057 14194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab94dffdb57447c853ebd56df142757" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33551 } }
I20260923 14:37:49.802585 14189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:37:49.803599 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ab94dffdb57447c853ebd56df142757. Latest consensus state: current_term: 1 leader_uuid: "9ab94dffdb57447c853ebd56df142757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab94dffdb57447c853ebd56df142757" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33551 } } }
I20260923 14:37:49.803705 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [sys.catalog]: This master's current role is: LEADER
I20260923 14:37:49.803725 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:49.792013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.2.218.62:33551)
0923 14:37:49.792352 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:49.792364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:49.792394 (+ 30us) server_negotiation.cc:407] Connection header received
0923 14:37:49.792948 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:49.792963 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:49.793227 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:49.793509 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:49.794248 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:49.795109 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:49.795854 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:49.796297 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:49.796442 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:49.798733 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:49.798737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:49.798747 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:37:49.798779 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:49.800505 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:49.800933 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:49.800938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:49.800942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:49.801101 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:49.801401 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:49.801404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:49.801406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:49.801841 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:49.802530 (+ 689us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:49.802657 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 14:37:49.802890 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":108,"threads_started":1}
I20260923 14:37:49.804070 14195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ab94dffdb57447c853ebd56df142757" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab94dffdb57447c853ebd56df142757" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33551 } } }
I20260923 14:37:49.804140 14203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:37:49.804144 14195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757 [sys.catalog]: This master's current role is: LEADER
I20260923 14:37:49.806798 14203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:37:49.807163 14189 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 14:37:49.815012 14203 catalog_manager.cc:1347] Generated new cluster ID: 0cc13cd3d95042bba9047a6d84020037
I20260923 14:37:49.815073 14203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:37:49.832365 14203 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:37:49.833532 14203 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:37:49.842343 14203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757: Generated new TSK 0
I20260923 14:37:49.843030 14203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:37:50.468855 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:44951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33551
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:39493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:37:50.573732 14234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:50.573994 14234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:50.574141 14234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:50.577682 14234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:50.577764 14234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:50.577864 14234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:37:50.582496 14234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:39493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.218.20:44951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:50.583516 14234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:50.584465 14234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:50.594736 14239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:50.594924 14242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:50.594969 14234 server_base.cc:1034] running on GCE node
W20260923 14:37:50.595194 14240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:50.595535 14234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:50.596099 14234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:50.597275 14234 hybrid_clock.cc:648] HybridClock initialized: now 1790174270597255 us; error 30 us; skew 500 ppm
Sep 23 14:37:50 dist-test-slave-zl47 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:37:50.599963 14234 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:37:50.600741 14234 webserver.cc:460] Webserver started at http://127.2.218.1:39805/ using document root <none> and password file <none>
I20260923 14:37:50.601259 14234 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:50.601344 14234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:50.601605 14234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:37:50 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:37:50.617202 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "100b179028944c00ab73aa06a414f8e4"
format_stamp: "Formatted at 2026-09-23 14:37:50 on dist-test-slave-zl47"
server_key: "178783b978d15708f9f6fa4ae9848312"
server_key_iv: "6f80a25dc438b2c1c86656671e818d15"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:50.618017 14234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "100b179028944c00ab73aa06a414f8e4"
format_stamp: "Formatted at 2026-09-23 14:37:50 on dist-test-slave-zl47"
server_key: "178783b978d15708f9f6fa4ae9848312"
server_key_iv: "6f80a25dc438b2c1c86656671e818d15"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:50.628286 14234 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.003s sys 0.001s
I20260923 14:37:50.650811 14251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:50.652241 14234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:37:50.652390 14234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "100b179028944c00ab73aa06a414f8e4"
format_stamp: "Formatted at 2026-09-23 14:37:50 on dist-test-slave-zl47"
server_key: "178783b978d15708f9f6fa4ae9848312"
server_key_iv: "6f80a25dc438b2c1c86656671e818d15"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:50.652547 14234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:50.674374 14234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:50.677471 14234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:50.677665 14234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:50.678328 14234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:50.679085 14234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:50.679149 14234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:50.679203 14234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:50.679243 14234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:50.690555 14234 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:33427
I20260923 14:37:50.690686 14364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:33427 every 8 connection(s)
I20260923 14:37:50.691546 14234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:37:50.694550 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 14234
I20260923 14:37:50.694659 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:37:50.694952 2920 external_mini_cluster.cc:1428] Setting key 3dada99352fb7d22d3dcd060c3aea938
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
Sep 23 14:37:50 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:37:50.709934 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:50.693797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33551 (local address 127.2.218.1:39343)
0923 14:37:50.694323 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:50.695375 (+ 1052us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:50.696430 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:50.698695 (+ 2265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:50.698704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:50.699023 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:50.699726 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:50.699744 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:50.700829 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:50.700833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:50.702142 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:50.702152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:50.702845 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:50.702851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:50.703059 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:50.703742 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:50.703772 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:50.705282 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:50.707190 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:50.707195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:50.707209 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:50.707507 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:50.707784 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:50.707787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:50.707788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:50.707927 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:50.708341 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:50.708346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:50.708597 (+ 251us) client_negotiation.cc:769] Sending connection context
0923 14:37:50.708744 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 14:37:50.709028 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":125,"threads_started":1}
I20260923 14:37:50.711089 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:50.693400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:39343 (local address 127.2.218.62:33551)
0923 14:37:50.698293 (+ 4893us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:50.698301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:50.698315 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:37:50.698352 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:50.698354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:50.698395 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:50.699874 (+ 1479us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:50.699979 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:50.700699 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:50.702301 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:50.702673 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:50.702761 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:50.705462 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:50.705465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:50.705467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:37:50.705486 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:50.707075 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:50.707618 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:50.707621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:50.707622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:50.707698 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:50.708029 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:50.708032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:50.708033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:50.708213 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:50.710773 (+ 2560us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:50.710836 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 14:37:50.710940 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4815,"thread_start_us":96,"threads_started":1}
I20260923 14:37:50.712118 14365 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33551
I20260923 14:37:50.712335 14365 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:50.712786 14365 heartbeater.cc:511] Master 127.2.218.62:33551 requested a full tablet report, sending...
I20260923 14:37:50.714376 14153 ts_manager.cc:194] Registered new tserver with Master: 100b179028944c00ab73aa06a414f8e4 (127.2.218.1:33427)
I20260923 14:37:50.715902 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:39343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:37:50.734588 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:44951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33551
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:39493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:37:50 dist-test-slave-zl47 krb5kdc[13676](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790174270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 14:37:50.880345 14373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:50.880613 14373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:50.880664 14373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:50.885778 14373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:50.885859 14373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:50.885958 14373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:37:50.890942 14373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:39493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.218.20:44951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:50.892014 14373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:50.893085 14373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:50.903697 14379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:50.903751 14382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:50.903923 14380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:50.904711 14373 server_base.cc:1034] running on GCE node
I20260923 14:37:50.905158 14373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:50.905828 14373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:50.907070 14373 hybrid_clock.cc:648] HybridClock initialized: now 1790174270907045 us; error 33 us; skew 500 ppm
Sep 23 14:37:50 dist-test-slave-zl47 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:37:50.909807 14373 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:37:50.910607 14373 webserver.cc:460] Webserver started at http://127.2.218.2:42753/ using document root <none> and password file <none>
I20260923 14:37:50.911084 14373 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:50.911135 14373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:50.911317 14373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:37:50 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:37:50.923645 14373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "3eaee9900b634904836812522ec9a79a"
format_stamp: "Formatted at 2026-09-23 14:37:50 on dist-test-slave-zl47"
server_key: "aeef68ef103d064182b90efe7bf864f1"
server_key_iv: "662335173a9a9f21fac68f6d5bb23239"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:50.924219 14373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "3eaee9900b634904836812522ec9a79a"
format_stamp: "Formatted at 2026-09-23 14:37:50 on dist-test-slave-zl47"
server_key: "aeef68ef103d064182b90efe7bf864f1"
server_key_iv: "662335173a9a9f21fac68f6d5bb23239"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:50.933012 14373 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.000s
I20260923 14:37:50.946229 14389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:50.947745 14373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:37:50.947889 14373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "3eaee9900b634904836812522ec9a79a"
format_stamp: "Formatted at 2026-09-23 14:37:50 on dist-test-slave-zl47"
server_key: "aeef68ef103d064182b90efe7bf864f1"
server_key_iv: "662335173a9a9f21fac68f6d5bb23239"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:50.948038 14373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:37:50.960358 14373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:50.962837 14373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:50.963027 14373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:50.963658 14373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:50.964632 14373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:50.964694 14373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:50.964742 14373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:50.964768 14373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:50.976613 14373 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:35191
I20260923 14:37:50.977635 14373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:37:50.980453 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 14373
I20260923 14:37:50.980568 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:37:50.980877 2920 external_mini_cluster.cc:1428] Setting key 84c542c53a172c6ba89324d451d24edb
Sep 23 14:37:50 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:37:50.991108 14502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:35191 every 8 connection(s)
I20260923 14:37:50.995702 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:50.980058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33551 (local address 127.2.218.2:51571)
0923 14:37:50.980547 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:50.980591 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:50.981922 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:50.982615 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:50.982624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:50.982925 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:50.983467 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:50.983483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:50.984554 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:50.984557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:50.985760 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:50.985769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:50.988466 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:50.988469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:50.988633 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:50.989280 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:50.989302 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:50.990934 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:50.992863 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:50.992868 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:50.992881 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:50.993153 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:50.993434 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:50.993437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:50.993438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:50.993611 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:50.994046 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:50.994051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:50.994346 (+ 295us) client_negotiation.cc:769] Sending connection context
0923 14:37:50.994532 (+ 186us) client_negotiation.cc:240] Negotiation successful
0923 14:37:50.994815 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
I20260923 14:37:50.996356 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:50.979695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:51571 (local address 127.2.218.62:33551)
0923 14:37:50.979833 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:50.979837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:50.980675 (+ 838us) server_negotiation.cc:407] Connection header received
0923 14:37:50.982171 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:50.982177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:50.982232 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:50.982320 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:50.983614 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:50.984392 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:50.985891 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:50.988225 (+ 2334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:50.988296 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:50.991100 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:50.991103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:50.991106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:37:50.991133 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:50.992731 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:50.993256 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:50.993259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:50.993260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:50.993335 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:50.993710 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:50.993713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:50.993715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:50.993901 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:50.993997 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:50.996072 (+ 2075us) server_negotiation.cc:299] Negotiation successful
0923 14:37:50.996204 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:37:50.997404 14503 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33551
I20260923 14:37:50.997658 14503 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
I20260923 14:37:50.998179 14503 heartbeater.cc:511] Master 127.2.218.62:33551 requested a full tablet report, sending...
I20260923 14:37:50.999230 14153 ts_manager.cc:194] Registered new tserver with Master: 3eaee9900b634904836812522ec9a79a (127.2.218.2:35191)
I20260923 14:37:50.999961 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:51571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:37:51.017122 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:44951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33551
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:39493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:37:51.144004 14508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:37:51.144291 14508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:37:51.144351 14508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:37:51.149274 14508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:37:51.149380 14508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:37:51.149502 14508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:37:51.156090 14508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:39493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.218.20:44951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:37:51.157245 14508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:37:51.158505 14508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:37:51.170068 14513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:51.170444 14514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:37:51.170720 14516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:37:51.171332 14508 server_base.cc:1034] running on GCE node
I20260923 14:37:51.171772 14508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:37:51.172435 14508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:37:51.173616 14508 hybrid_clock.cc:648] HybridClock initialized: now 1790174271173596 us; error 33 us; skew 500 ppm
Sep 23 14:37:51 dist-test-slave-zl47 krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:37:51.176961 14508 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:37:51.177877 14508 webserver.cc:460] Webserver started at http://127.2.218.3:32849/ using document root <none> and password file <none>
I20260923 14:37:51.178519 14508 fs_manager.cc:359] Metadata directory not provided
I20260923 14:37:51.178592 14508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:37:51.178781 14508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:37:51 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:37:51.192588 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "28ef4fa0591a4ba096d60317b2dd4e18"
format_stamp: "Formatted at 2026-09-23 14:37:51 on dist-test-slave-zl47"
server_key: "807eff91f7592c305e598fa46a66501f"
server_key_iv: "2f325211a18ed0712f1d309343dfa2e6"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:51.193445 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "28ef4fa0591a4ba096d60317b2dd4e18"
format_stamp: "Formatted at 2026-09-23 14:37:51 on dist-test-slave-zl47"
server_key: "807eff91f7592c305e598fa46a66501f"
server_key_iv: "2f325211a18ed0712f1d309343dfa2e6"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:51.199393 14508 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.000s
I20260923 14:37:51.221179 14523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:51.222738 14508 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.001s
I20260923 14:37:51.222841 14508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "28ef4fa0591a4ba096d60317b2dd4e18"
format_stamp: "Formatted at 2026-09-23 14:37:51 on dist-test-slave-zl47"
server_key: "807eff91f7592c305e598fa46a66501f"
server_key_iv: "2f325211a18ed0712f1d309343dfa2e6"
server_key_version: "kuduclusterkey@0"
I20260923 14:37:51.222937 14508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 23 14:37:51 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:37:51.245458 14508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:37:51.249872 14508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:37:51.250048 14508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:37:51.250649 14508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:37:51.251503 14508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:37:51.251579 14508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:51.251662 14508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:37:51.251709 14508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:37:51.263108 14508 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:38323
I20260923 14:37:51.264148 14508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:37:51.266790 14636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:38323 every 8 connection(s)
I20260923 14:37:51.267326 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 14508
I20260923 14:37:51.267427 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:37:51.267695 2920 external_mini_cluster.cc:1428] Setting key aa54d5bbdd73061a7473a58e404c7a35
Sep 23 14:37:51 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174271, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:37:51.284873 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.273687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:58229 (local address 127.2.218.62:33551)
0923 14:37:51.273793 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:51.273798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:51.273812 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:37:51.274798 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:51.274802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:51.274851 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:51.274921 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:51.276072 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.276806 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.278266 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.278579 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.278645 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.281317 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:51.281321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:51.281323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:37:51.281346 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:51.282844 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:51.283368 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:51.283371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:51.283373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:51.283447 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:51.283797 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:51.283800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:51.283801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:51.283989 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:51.284054 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:51.284626 (+ 572us) server_negotiation.cc:299] Negotiation successful
0923 14:37:51.284731 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:37:51.286146 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.272763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33551 (local address 127.2.218.3:58229)
0923 14:37:51.273247 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:51.273297 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:51.274532 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:51.275104 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:51.275112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:51.275403 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:37:51.275927 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.275942 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.276941 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.276944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.278157 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.278166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.278807 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.278811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.278971 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.279528 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:37:51.279546 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:37:51.281157 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:37:51.282963 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:51.282968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:51.282979 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:51.283249 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:51.283547 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:37:51.283549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:37:51.283551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:37:51.283689 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:37:51.284137 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:37:51.284142 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:37:51.284401 (+ 259us) client_negotiation.cc:769] Sending connection context
0923 14:37:51.285071 (+ 670us) client_negotiation.cc:240] Negotiation successful
0923 14:37:51.285338 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20260923 14:37:51.288586 14637 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33551
I20260923 14:37:51.288822 14637 heartbeater.cc:464] Registering TS with master...
I20260923 14:37:51.289288 14637 heartbeater.cc:511] Master 127.2.218.62:33551 requested a full tablet report, sending...
I20260923 14:37:51.290205 14153 ts_manager.cc:194] Registered new tserver with Master: 28ef4fa0591a4ba096d60317b2dd4e18 (127.2.218.3:38323)
I20260923 14:37:51.290855 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:58229
I20260923 14:37:51.292328 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:37:51.304137 14367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.295763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.2.218.62:33551)
0923 14:37:51.295854 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:51.295857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:51.296402 (+ 545us) server_negotiation.cc:407] Connection header received
0923 14:37:51.296536 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:51.296538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:51.296572 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:51.296609 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:37:51.297417 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.298080 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.299189 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.299458 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.299523 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.301009 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:37:51.301012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:37:51.301014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:37:51.301035 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:37:51.302504 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:51.302945 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:51.302948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:51.302949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:51.303016 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:37:51.303254 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:37:51.303257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:37:51.303258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:37:51.303442 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:37:51.303531 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:51.303923 (+ 392us) server_negotiation.cc:299] Negotiation successful
0923 14:37:51.304018 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:37:51.307355 14153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:37:51.310475 14153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:37:51.324357 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.317599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:33427 (local address 127.0.0.1:54838)
0923 14:37:51.318549 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:51.318589 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:51.318697 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:51.320068 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:51.320073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:51.320090 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:37:51.320217 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.320224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.321469 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.321472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.322781 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.322789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.323859 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.323862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.324009 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.324048 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 14:37:51.324134 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 14:37:51.324230 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":18,"thread_start_us":75,"threads_started":1}
I20260923 14:37:51.324870 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.318597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.2.218.1:33427)
0923 14:37:51.319719 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:51.319723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:51.319741 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:37:51.319795 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:51.319804 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:51.319934 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:51.320067 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:37:51.320347 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.321372 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.322899 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.323616 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.323701 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.323811 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:51.324657 (+ 846us) server_negotiation.cc:299] Negotiation successful
0923 14:37:51.324753 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1035,"thread_start_us":63,"threads_started":1}
I20260923 14:37:51.326468 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.318949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:35191 (local address 127.0.0.1:59928)
0923 14:37:51.320489 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:51.320498 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:51.320561 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:51.321241 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:51.321243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:51.321256 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:37:51.321336 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.321341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.323036 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.323039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.324009 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.324016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.326152 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.326155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.326237 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.326256 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 14:37:51.326309 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 14:37:51.326363 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1494,"mutex_wait_us":33,"thread_start_us":84,"threads_started":1}
I20260923 14:37:51.327026 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.319101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.2.218.2:35191)
0923 14:37:51.320750 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:51.320759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:51.320777 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:37:51.320845 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:51.320849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:51.321009 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:51.321134 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:37:51.321448 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.322938 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.324140 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.325996 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.326083 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.326789 (+ 706us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:51.326844 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 14:37:51.326913 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566,"thread_start_us":125,"threads_started":1}
I20260923 14:37:51.327641 14299 tablet_service.cc:1467] Processing CreateTablet for tablet 876d0b9f56f94ed7a33be2f2362ea73f (DEFAULT_TABLE table=test-table [id=4220504b06c8416fbdbbc353986bda6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:37:51.328631 14299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876d0b9f56f94ed7a33be2f2362ea73f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:37:51.331923 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.318266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:38323 (local address 127.0.0.1:54578)
0923 14:37:51.320265 (+ 1999us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:51.320278 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:51.320347 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:51.327549 (+ 7202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:51.327551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:51.327567 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:37:51.327638 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.327644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.329028 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.329031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.329896 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.329905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.331642 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.331645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.331714 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.331732 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 14:37:51.331769 (+ 37us) client_negotiation.cc:240] Negotiation successful
0923 14:37:51.331828 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1941,"thread_start_us":75,"threads_started":1}
I20260923 14:37:51.332590 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.318399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.2.218.3:38323)
0923 14:37:51.327156 (+ 8757us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:51.327167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:51.327189 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:37:51.327252 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:51.327257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:51.327409 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:51.327558 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:37:51.327762 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.328792 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.330047 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.331527 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.332261 (+ 734us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.332375 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:51.332415 (+ 40us) server_negotiation.cc:299] Negotiation successful
0923 14:37:51.332479 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8666,"thread_start_us":164,"threads_started":1}
I20260923 14:37:51.332649 14436 tablet_service.cc:1467] Processing CreateTablet for tablet 876d0b9f56f94ed7a33be2f2362ea73f (DEFAULT_TABLE table=test-table [id=4220504b06c8416fbdbbc353986bda6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:37:51.333564 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876d0b9f56f94ed7a33be2f2362ea73f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:37:51.334889 14560 tablet_service.cc:1467] Processing CreateTablet for tablet 876d0b9f56f94ed7a33be2f2362ea73f (DEFAULT_TABLE table=test-table [id=4220504b06c8416fbdbbc353986bda6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:37:51.335742 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 876d0b9f56f94ed7a33be2f2362ea73f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:37:51.341382 14655 tablet_bootstrap.cc:492] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4: Bootstrap starting.
I20260923 14:37:51.343241 14657 tablet_bootstrap.cc:492] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: Bootstrap starting.
I20260923 14:37:51.344218 14655 tablet_bootstrap.cc:654] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4: Neither blocks nor log segments found. Creating new log.
I20260923 14:37:51.345181 14655 log.cc:824] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4: Log is configured to *not* fsync() on all Append() calls
I20260923 14:37:51.345557 14657 tablet_bootstrap.cc:654] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: Neither blocks nor log segments found. Creating new log.
I20260923 14:37:51.345932 14656 tablet_bootstrap.cc:492] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: Bootstrap starting.
I20260923 14:37:51.346297 14657 log.cc:824] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: Log is configured to *not* fsync() on all Append() calls
I20260923 14:37:51.348385 14657 tablet_bootstrap.cc:492] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: No bootstrap required, opened a new log
I20260923 14:37:51.348413 14655 tablet_bootstrap.cc:492] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4: No bootstrap required, opened a new log
I20260923 14:37:51.348594 14657 ts_tablet_manager.cc:1397] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260923 14:37:51.348691 14655 ts_tablet_manager.cc:1397] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260923 14:37:51.352300 14657 raft_consensus.cc:348] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 } } peers { permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 } } peers { permanent_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38323 } }
I20260923 14:37:51.352603 14657 raft_consensus.cc:374] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:37:51.352682 14657 raft_consensus.cc:729] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ef4fa0591a4ba096d60317b2dd4e18, State: Initialized, Role: FOLLOWER
I20260923 14:37:51.353154 14657 consensus_queue.cc:260] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 } } peers { permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 } } peers { permanent_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38323 } }
I20260923 14:37:51.353314 14656 tablet_bootstrap.cc:654] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: Neither blocks nor log segments found. Creating new log.
I20260923 14:37:51.354115 14657 ts_tablet_manager.cc:1428] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260923 14:37:51.354475 14637 heartbeater.cc:503] Master 127.2.218.62:33551 was elected leader, sending a full tablet report...
I20260923 14:37:51.354888 14655 raft_consensus.cc:348] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 } } peers { permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 } } peers { permanent_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38323 } }
I20260923 14:37:51.355118 14655 raft_consensus.cc:374] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:37:51.355177 14655 raft_consensus.cc:729] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100b179028944c00ab73aa06a414f8e4, State: Initialized, Role: FOLLOWER
I20260923 14:37:51.359076 14656 log.cc:824] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:37:51.359774 14655 consensus_queue.cc:260] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 } } peers { permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 } } peers { permanent_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38323 } }
I20260923 14:37:51.360630 14365 heartbeater.cc:503] Master 127.2.218.62:33551 was elected leader, sending a full tablet report...
I20260923 14:37:51.362218 14655 ts_tablet_manager.cc:1428] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260923 14:37:51.362710 14656 tablet_bootstrap.cc:492] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: No bootstrap required, opened a new log
I20260923 14:37:51.363095 14656 ts_tablet_manager.cc:1397] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260923 14:37:51.367301 14656 raft_consensus.cc:348] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 } } peers { permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 } } peers { permanent_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38323 } }
I20260923 14:37:51.367774 14656 raft_consensus.cc:374] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:37:51.368093 14656 raft_consensus.cc:729] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eaee9900b634904836812522ec9a79a, State: Initialized, Role: FOLLOWER
I20260923 14:37:51.368553 14656 consensus_queue.cc:260] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 } } peers { permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 } } peers { permanent_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38323 } }
I20260923 14:37:51.369163 14503 heartbeater.cc:503] Master 127.2.218.62:33551 was elected leader, sending a full tablet report...
I20260923 14:37:51.371052 14656 ts_tablet_manager.cc:1428] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260923 14:37:51.443188 14366 tablet.cc:2369] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 14:37:51 dist-test-slave-zl47 krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174270, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
W20260923 14:37:51.492023 14504 tablet.cc:2369] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:37:51.518469 14638 tablet.cc:2369] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:37:51.639086 14660 raft_consensus.cc:482] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:37:51.639324 14660 raft_consensus.cc:504] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 } } peers { permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 } } peers { permanent_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38323 } }
I20260923 14:37:51.640673 14660 leader_election.cc:288] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100b179028944c00ab73aa06a414f8e4 (127.2.218.1:33427), 3eaee9900b634904836812522ec9a79a (127.2.218.2:35191)
I20260923 14:37:51.645833 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.640823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:33427 (local address 127.2.218.3:55305)
0923 14:37:51.641444 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:51.641458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:51.641540 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:51.642384 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:51.642386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:51.642403 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:37:51.642516 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.642522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.643479 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.643482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.644568 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.644578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.645473 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.645476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.645560 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.645578 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 14:37:51.645659 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 14:37:51.645716 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549}
I20260923 14:37:51.646816 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.641812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:55305 (local address 127.2.218.1:33427)
0923 14:37:51.642064 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:51.642068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:51.642080 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:37:51.642149 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:51.642151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:51.642197 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:51.642278 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:37:51.642644 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.643357 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.644741 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.645276 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.645364 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.645422 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:51.646634 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0923 14:37:51.646702 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260923 14:37:51.647830 14319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876d0b9f56f94ed7a33be2f2362ea73f" candidate_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100b179028944c00ab73aa06a414f8e4" is_pre_election: true
I20260923 14:37:51.648115 14319 raft_consensus.cc:2454] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28ef4fa0591a4ba096d60317b2dd4e18 in term 0.
I20260923 14:37:51.648708 14527 leader_election.cc:302] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100b179028944c00ab73aa06a414f8e4, 28ef4fa0591a4ba096d60317b2dd4e18; no voters:
I20260923 14:37:51.648986 14660 raft_consensus.cc:2781] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:37:51.649051 14660 raft_consensus.cc:482] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:37:51.649087 14660 raft_consensus.cc:3037] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:37:51.650630 14660 raft_consensus.cc:504] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 } } peers { permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 } } peers { permanent_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38323 } }
I20260923 14:37:51.650982 14660 leader_election.cc:288] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [CANDIDATE]: Term 1 election: Requested vote from peers 100b179028944c00ab73aa06a414f8e4 (127.2.218.1:33427), 3eaee9900b634904836812522ec9a79a (127.2.218.2:35191)
I20260923 14:37:51.651597 14319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876d0b9f56f94ed7a33be2f2362ea73f" candidate_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100b179028944c00ab73aa06a414f8e4"
I20260923 14:37:51.651716 14319 raft_consensus.cc:3037] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:37:51.652885 14319 raft_consensus.cc:2454] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28ef4fa0591a4ba096d60317b2dd4e18 in term 1.
I20260923 14:37:51.653268 14527 leader_election.cc:302] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100b179028944c00ab73aa06a414f8e4, 28ef4fa0591a4ba096d60317b2dd4e18; no voters:
I20260923 14:37:51.653466 14660 raft_consensus.cc:2781] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:37:51.653628 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.641076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:35191 (local address 127.2.218.3:51219)
0923 14:37:51.645961 (+ 4885us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:51.645973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:37:51.646041 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:37:51.646453 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:37:51.646456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:37:51.646470 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:37:51.646538 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.646544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.651479 (+ 4935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.651485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.652400 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:37:51.652409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.653284 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.653289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:37:51.653361 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.653379 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 14:37:51.653442 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 14:37:51.653499 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4841,"thread_start_us":96,"threads_started":1}
I20260923 14:37:51.653637 14660 raft_consensus.cc:686] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 1 LEADER]: Becoming Leader. State: Replica: 28ef4fa0591a4ba096d60317b2dd4e18, State: Running, Role: LEADER
I20260923 14:37:51.653959 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.641932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:51219 (local address 127.2.218.2:35191)
0923 14:37:51.642039 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:51.642043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:51.646257 (+ 4214us) server_negotiation.cc:407] Connection header received
0923 14:37:51.646310 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:51.646313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:51.646353 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:51.646448 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:37:51.646649 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.647270 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.652532 (+ 5262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.653085 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.653178 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.653236 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:51.653764 (+ 528us) server_negotiation.cc:299] Negotiation successful
0923 14:37:51.653846 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:37:51.654214 14660 consensus_queue.cc:237] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 } } peers { permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 } } peers { permanent_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38323 } }
I20260923 14:37:51.654466 14457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876d0b9f56f94ed7a33be2f2362ea73f" candidate_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eaee9900b634904836812522ec9a79a" is_pre_election: true
I20260923 14:37:51.654609 14456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "876d0b9f56f94ed7a33be2f2362ea73f" candidate_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eaee9900b634904836812522ec9a79a"
I20260923 14:37:51.654709 14457 raft_consensus.cc:2454] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28ef4fa0591a4ba096d60317b2dd4e18 in term 0.
I20260923 14:37:51.657095 14153 catalog_manager.cc:5495] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28ef4fa0591a4ba096d60317b2dd4e18 (127.2.218.3). New cstate: current_term: 1 leader_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28ef4fa0591a4ba096d60317b2dd4e18" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 38323 } health_report { overall_health: HEALTHY } } }
I20260923 14:37:51.685626 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.681326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.2.218.3:38323)
0923 14:37:51.681456 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:37:51.681461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:37:51.681502 (+ 41us) server_negotiation.cc:407] Connection header received
0923 14:37:51.681640 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:37:51.681642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:37:51.681698 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:37:51.681813 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:37:51.682065 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.682559 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.683614 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:37:51.683939 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:37:51.684056 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:37:51.684443 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:37:51.684952 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:37:51.685060 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 14:37:51.685423 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 14:37:51.685476 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 14:37:51.693940 14319 raft_consensus.cc:1261] T 876d0b9f56f94ed7a33be2f2362ea73f P 100b179028944c00ab73aa06a414f8e4 [term 1 FOLLOWER]: Refusing update from remote peer 28ef4fa0591a4ba096d60317b2dd4e18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:37:51.694710 14660 consensus_queue.cc:1035] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "100b179028944c00ab73aa06a414f8e4" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:37:51.698577 14456 raft_consensus.cc:3037] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:37:51.699970 14456 raft_consensus.cc:1261] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a [term 1 FOLLOWER]: Refusing update from remote peer 28ef4fa0591a4ba096d60317b2dd4e18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:37:51.700657 14660 consensus_queue.cc:1035] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eaee9900b634904836812522ec9a79a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:37:51.701283 14672 mvcc.cc:205] Tried to move back new op lower bound from 7332553816841248768 to 7332553816698318848. Current Snapshot: MvccSnapshot[applied={T|T < 7332553816841248768}]
I20260923 14:37:51.712095 14677 mvcc.cc:205] Tried to move back new op lower bound from 7332553816841248768 to 7332553816698318848. Current Snapshot: MvccSnapshot[applied={T|T < 7332553816841248768}]
I20260923 14:37:51.716840 14153 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:37:51.716997 14153 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:37:51.719035 14153 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ab94dffdb57447c853ebd56df142757: Sending DeleteTablet for 3 replicas of tablet 876d0b9f56f94ed7a33be2f2362ea73f
I20260923 14:37:51.719623 14299 tablet_service.cc:1514] Processing DeleteTablet for tablet 876d0b9f56f94ed7a33be2f2362ea73f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:37:51 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:54838
I20260923 14:37:51.719822 14436 tablet_service.cc:1514] Processing DeleteTablet for tablet 876d0b9f56f94ed7a33be2f2362ea73f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:37:51 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:59928
I20260923 14:37:51.719985 14560 tablet_service.cc:1514] Processing DeleteTablet for tablet 876d0b9f56f94ed7a33be2f2362ea73f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:37:51 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:54578
I20260923 14:37:51.720410 14686 tablet_replica.cc:330] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: stopping tablet replica
I20260923 14:37:51.720661 14686 raft_consensus.cc:2229] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:37:51.720809 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 14234
I20260923 14:37:51.721474 14687 tablet_replica.cc:330] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: stopping tablet replica
I20260923 14:37:51.721760 14687 raft_consensus.cc:2229] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:37:51.722121 14687 raft_consensus.cc:2258] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:37:51.722378 14686 raft_consensus.cc:2258] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:37:51.724704 14687 ts_tablet_manager.cc:1905] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:37:51.727878 14686 ts_tablet_manager.cc:1905] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 14:37:51.729699 14140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:33427 (error 108)
I20260923 14:37:51.730480 14686 ts_tablet_manager.cc:1918] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:37:51.730563 14686 log.cc:1196] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/876d0b9f56f94ed7a33be2f2362ea73f
I20260923 14:37:51.730630 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.730361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:33427 (local address 127.0.0.1:54852)
0923 14:37:51.730484 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:51.730550 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:33427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 14:37:51.730767 14140 catalog_manager.cc:4584] TS 100b179028944c00ab73aa06a414f8e4 (127.2.218.1:33427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 876d0b9f56f94ed7a33be2f2362ea73f: Network error: Client connection negotiation failed: client connection to 127.2.218.1:33427: connect: Connection refused (error 111)
I20260923 14:37:51.730849 14686 ts_tablet_manager.cc:1939] T 876d0b9f56f94ed7a33be2f2362ea73f P 3eaee9900b634904836812522ec9a79a: Deleting consensus metadata
I20260923 14:37:51.730854 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 14373
I20260923 14:37:51.732234 14687 ts_tablet_manager.cc:1918] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:37:51.732308 14687 log.cc:1196] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/876d0b9f56f94ed7a33be2f2362ea73f
I20260923 14:37:51.732584 14687 ts_tablet_manager.cc:1939] T 876d0b9f56f94ed7a33be2f2362ea73f P 28ef4fa0591a4ba096d60317b2dd4e18: Deleting consensus metadata
I20260923 14:37:51.733515 14139 catalog_manager.cc:4841] TS 28ef4fa0591a4ba096d60317b2dd4e18 (127.2.218.3:38323): tablet 876d0b9f56f94ed7a33be2f2362ea73f (table test-table [id=4220504b06c8416fbdbbc353986bda6f]) successfully deleted
I20260923 14:37:51.737133 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 14508
I20260923 14:37:51.737512 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:37:51.737309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:35191 (local address 127.0.0.1:59934)
0923 14:37:51.737424 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 14:37:51.737458 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.2:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 14:37:51.742308 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 14121
I20260923 14:37:51.747694 2920 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 14:37:52.081539 2920 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 14:37:52.081643 2920 mini_ranger.cc:66] Stopping Ranger...
I20260923 14:37:52.415393 2920 mini_ranger.cc:68] Stopped Ranger
2026-09-23 14:37:52.415 UTC [13706] LOG: received smart shutdown request
2026-09-23 14:37:52.419 UTC [13706] LOG: background worker "logical replication launcher" (PID 13715) exited with exit code 1
2026-09-23 14:37:52.419 UTC [13710] LOG: shutting down
2026-09-23 14:37:52.556 UTC [13706] LOG: database system is shut down
2026-09-23T14:37:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (41852 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:37:52 dist-test-slave-zl47 krb5kdc[14699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:37:52 dist-test-slave-zl47 krb5kdc[14699](info): set up 2 sockets
Sep 23 14:37:52 dist-test-slave-zl47 krb5kdc[14699](info): commencing operation
krb5kdc: starting...
W20260923 14:37:53.570289 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.874s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:37:53 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:37:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:37:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/rangeradmin_127.2.218.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/rangeradmin_127.2.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/rangerlookup_127.2.218.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/rangerlookup_127.2.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/HTTP_127.2.218.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/HTTP_127.2.218.20@KRBTEST.COM.keytab.
I20260923 14:37:53.705577 2920 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task3TRGqf/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/postgres -l logfile start
2026-09-23 14:37:55.813 UTC [14729] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 14:37:55.813 UTC [14729] LOG: listening on IPv4 address "127.2.218.20", port 35945
2026-09-23 14:37:55.816 UTC [14729] LOG: listening on Unix socket "/tmp/.s.PGSQL.35945"
2026-09-23 14:37:55.833 UTC [14732] LOG: database system was shut down at 2026-09-23 14:37:54 UTC
2026-09-23 14:37:55.836 UTC [14729] LOG: database system is ready to accept connections
I20260923 14:37:56.653951 2920 mini_postgres.cc:96] Postgres bound to 35945
2026-09-23 14:37:56.659 UTC [14740] FATAL: database "slave" does not exist
127.2.218.20:35945 - accepting connections
I20260923 14:37:56.659989 2920 mini_ranger.cc:161] Starting Ranger...
I20260923 14:37:56.672590 2920 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 14:37:57.064826 2920 mini_ranger.cc:87] Created ranger Postgres database
I20260923 14:37:57.064917 2920 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-admin
2026-09-23 14:37:57,294 [I] DB FLAVOR :POSTGRES
2026-09-23 14:37:57,295 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 14:37:57,295 [I] Checking connection..
2026-09-23 14:37:57,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 14:37:57,492 [I] Checking connection passed.
2026-09-23 14:37:57,492 [I] --------- Verifying version history table ---------
2026-09-23 14:37:57,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:37:57,683 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 14:37:57,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:37:57,894 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 14:37:57,894 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 14:37:57,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 14:37:58.102 UTC [14804] WARNING: there is no transaction in progress
2026-09-23 14:37:58,107 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 14:37:58,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 14:37:58,300 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 14:37:58,300 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 14:37:58,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 14:37:58,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 14:37:58,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zl47.c.gcp-upstream.internal','N') ;"
2026-09-23 14:37:58,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 14:37:59,117 [I] Table x_portal_user does not exist in database ranger
2026-09-23 14:37:59,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 14:37:59,327 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 14:37:59,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 14:37:59,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 14:37:59,728 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 14:37:59,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 14:38:00.516 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.530 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.544 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.557 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.570 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.584 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.630 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.635 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.647 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.656 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.666 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.678 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.686 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.690 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.696 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:00.701 UTC [14912] WARNING: there is no transaction in progress
2026-09-23 14:38:01,238 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 14:38:01,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 14:38:01,448 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 14:38:01,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 14:38:01,667 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 14:38:01,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 14:38:01,897 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 14:38:01,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 14:38:02,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 14:38:02,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zl47.c.gcp-upstream.internal';"
2026-09-23 14:38:02,510 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 14:38:02,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 14:38:02,705 [I] DB_PATCHES have already been applied
I20260923 14:38:02.710322 2920 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-admin:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task3TRGqf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 14:38:02.710376 2920 mini_ranger.cc:193] Using host: 127.2.218.20
I20260923 14:38:02.713340 2920 mini_ranger.cc:227] Ranger admin URL: http://127.2.218.20:36913
Sep 23, 2026 2:38:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 2:38:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 2:38:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 2:38:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 2:38:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 2:38:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 2:38:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 14:38:03.671727 2920 mini_ranger.cc:160] Time spent starting Ranger: real 7.012s user 0.000s sys 0.008s
Sep 23, 2026 2:38:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 2:38:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 2:38:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 2:38:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 2:38:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 14:38:22.536652 2920 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.218.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.218.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/rangerkms_127.2.218.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.218.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/rangerkms_127.2.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.218.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.218.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.218.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/HTTP_127.2.218.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.218.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/HTTP_127.2.218.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 14:38:22 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174302, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 14:38:22.613162 2920 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 14:38:22.621541 2920 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 14:38:23.341264 2920 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 14:38:23.341351 2920 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-kms
I20260923 14:38:23.341367 2920 mini_ranger_kms.cc:227] Using postgres at 127.2.218.20:35945
2026-09-23 14:38:23,492 [I] DB FLAVOR :POSTGRES
2026-09-23 14:38:23,493 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 14:38:23,493 [I] Checking connection
2026-09-23 14:38:23,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 14:38:23,710 [I] connection success
2026-09-23 14:38:23,710 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 14:38:23,710 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 14:38:23,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 14:38:23,911 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 14:38:23,911 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 14:38:23,911 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 14:38:23,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.218.20:35945/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 14:38:24,145 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 14:38:24.514671 2920 mini_ranger_kms.cc:310] Created kms service
I20260923 14:38:24.696295 2920 mini_ranger_kms.cc:326] Created kudu user
I20260923 14:38:24.742296 2920 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 14:38:24 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174302, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:25.093271 2920 mini_ranger_kms.cc:384] Added ranger policy
I20260923 14:38:25.093434 2920 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-kms:/tmp/dist-test-task3TRGqf/build/debug/bin/postgresql.jar:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task3TRGqf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 14:38:25.093457 2920 mini_ranger_kms.cc:240] Using host: 127.2.218.20
I20260923 14:38:25.096810 2920 mini_ranger_kms.cc:276] Ranger KMS PID: 15092
I20260923 14:38:25.096884 2920 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.218.20:55795
Sep 23, 2026 2:38:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 2:38:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 2:38:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 2:38:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 2:38:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 2:38:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 14:38:27.066511 2920 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.453s user 0.002s sys 0.135s
I20260923 14:38:27.066640 2920 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 14:38:27.066700 2920 mini_ranger_kms.cc:401] 127.2.218.20:55795/kms/v1/keys
Sep 23 14:38:27 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790174307, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.218.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 14:38:27 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174307, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.218.20@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
Sep 23 14:38:27 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174307, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.218.20@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
Sep 23 14:38:29 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:38:29.814631 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36987
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:36987
--ranger_config_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:55795/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:29.908859 15142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:29.909034 15142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:29.909065 15142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:29.912561 15142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:38:29.912602 15142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:29.912618 15142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:29.912631 15142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:38:29.912690 15142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:38:29.916989 15142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.218.20:55795/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:36987
--ranger_config_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:29.917704 15142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:29.918507 15142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:29.925359 15148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:29.925340 15147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:29.925499 15150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:29.925598 15142 server_base.cc:1034] running on GCE node
I20260923 14:38:29.926083 15142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:29.927169 15142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:29.928356 15142 hybrid_clock.cc:648] HybridClock initialized: now 1790174309928335 us; error 37 us; skew 500 ppm
Sep 23 14:38:29 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:29.934024 15142 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:38:29.934844 15142 webserver.cc:460] Webserver started at http://127.2.218.62:39731/ using document root <none> and password file <none>
I20260923 14:38:29.935276 15142 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:29.935326 15142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:29.935508 15142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:38:29 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174309, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:29.975219 15142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "353a68d0ee7c4d5a843bc4a0020b504c"
format_stamp: "Formatted at 2026-09-23 14:38:29 on dist-test-slave-zl47"
server_key: "82d180d059b6da102ea8b04701396d41"
server_key_iv: "15d20a529b965b6b37ef3bb48fbbfb3a"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:29.975709 15142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "353a68d0ee7c4d5a843bc4a0020b504c"
format_stamp: "Formatted at 2026-09-23 14:38:29 on dist-test-slave-zl47"
server_key: "82d180d059b6da102ea8b04701396d41"
server_key_iv: "15d20a529b965b6b37ef3bb48fbbfb3a"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:29.978852 15142 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:38:29.989523 15157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:29.990666 15142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:38:29.990792 15142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "353a68d0ee7c4d5a843bc4a0020b504c"
format_stamp: "Formatted at 2026-09-23 14:38:29 on dist-test-slave-zl47"
server_key: "82d180d059b6da102ea8b04701396d41"
server_key_iv: "15d20a529b965b6b37ef3bb48fbbfb3a"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:29.990898 15142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:30.006271 15142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:30.008879 15142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:30.009099 15142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:30.017758 15142 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:36987
I20260923 14:38:30.017762 15209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:36987 every 8 connection(s)
I20260923 14:38:30.018769 15142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:38:30.019898 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 15142
I20260923 14:38:30.020035 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:38:30.020294 2920 external_mini_cluster.cc:1428] Setting key a8fbaafa739cf03a04829a6d2b13476b
I20260923 14:38:30.022078 15210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 14:38:30 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:30.029202 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: Bootstrap starting.
I20260923 14:38:30.031731 15210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:30.032969 15210 log.cc:824] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:30.033303 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:30.021671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.2.218.62:36987)
0923 14:38:30.022029 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:30.022048 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:30.022087 (+ 39us) server_negotiation.cc:407] Connection header received
0923 14:38:30.022771 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:30.022787 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:30.023092 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:30.023420 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:30.024132 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:30.025107 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:30.026010 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:30.026491 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:30.026668 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:30.029148 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:30.029153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:30.029164 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:30.029188 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:30.030803 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:30.031201 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:30.031210 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:30.031214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:30.031309 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:30.031567 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:30.031571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:30.031573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:30.031952 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:30.032118 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:30.032402 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 14:38:30.032652 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":120,"threads_started":1}
I20260923 14:38:30.039208 15210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: No bootstrap required, opened a new log
I20260923 14:38:30.042409 15210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } }
I20260923 14:38:30.042615 15210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:30.042678 15210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353a68d0ee7c4d5a843bc4a0020b504c, State: Initialized, Role: FOLLOWER
I20260923 14:38:30.043053 15210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } }
I20260923 14:38:30.043188 15210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:38:30.043242 15210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:38:30.043326 15210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:30.044016 15210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } }
I20260923 14:38:30.044364 15210 leader_election.cc:302] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 353a68d0ee7c4d5a843bc4a0020b504c; no voters:
I20260923 14:38:30.044687 15210 leader_election.cc:288] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:38:30.045148 15216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:38:30.045357 15216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 1 LEADER]: Becoming Leader. State: Replica: 353a68d0ee7c4d5a843bc4a0020b504c, State: Running, Role: LEADER
I20260923 14:38:30.045698 15216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } }
I20260923 14:38:30.045763 15210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:38:30.047959 15210 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 14:38:30.050721 15217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } } }
I20260923 14:38:30.051092 15217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:30.051856 15231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:38:30.053896 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353a68d0ee7c4d5a843bc4a0020b504c. Latest consensus state: current_term: 1 leader_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } } }
I20260923 14:38:30.054086 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:30.056454 15231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:38:30.063637 15231 catalog_manager.cc:1347] Generated new cluster ID: 341767961fc6479ea56c4c98c76873d1
I20260923 14:38:30.063683 15231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:38:30.071877 15231 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:38:30.072968 15231 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:38:30.079176 15231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: Generated new TSK 0
I20260923 14:38:30.080178 15231 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:38:30.753103 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:55795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36987
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:30.885875 15257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:30.886190 15257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:30.886318 15257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:30.891592 15257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:30.891652 15257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:30.891721 15257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:38:30.898058 15257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.218.20:55795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:30.899320 15257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:30.900456 15257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:30.911113 15265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:30.911340 15257 server_base.cc:1034] running on GCE node
W20260923 14:38:30.911556 15263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:30.911648 15262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:30.912156 15257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:30.912770 15257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:30.913954 15257 hybrid_clock.cc:648] HybridClock initialized: now 1790174310913937 us; error 31 us; skew 500 ppm
Sep 23 14:38:30 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:30.917013 15257 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:38:30.917905 15257 webserver.cc:460] Webserver started at http://127.2.218.1:40273/ using document root <none> and password file <none>
I20260923 14:38:30.918565 15257 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:30.918648 15257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:30.918906 15257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:38:30 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:30.935997 15257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "371f0e39086c4563b09e239daf4bee7a"
format_stamp: "Formatted at 2026-09-23 14:38:30 on dist-test-slave-zl47"
server_key: "8d69c3a96e4903beeb537e2fff060b29"
server_key_iv: "0069b9cfb0983e8fcde0360a4cfd11aa"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:30.936558 15257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "371f0e39086c4563b09e239daf4bee7a"
format_stamp: "Formatted at 2026-09-23 14:38:30 on dist-test-slave-zl47"
server_key: "8d69c3a96e4903beeb537e2fff060b29"
server_key_iv: "0069b9cfb0983e8fcde0360a4cfd11aa"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:30.942637 15257 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 14:38:30.960929 15274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:30.962589 15257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:38:30.962787 15257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "371f0e39086c4563b09e239daf4bee7a"
format_stamp: "Formatted at 2026-09-23 14:38:30 on dist-test-slave-zl47"
server_key: "8d69c3a96e4903beeb537e2fff060b29"
server_key_iv: "0069b9cfb0983e8fcde0360a4cfd11aa"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:30.962996 15257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:30.990222 15257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:30.993324 15257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:30.993525 15257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:30.994195 15257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:30.995208 15257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:30.995311 15257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:30.995433 15257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:30.995519 15257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:31.004772 15257 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:36341
I20260923 14:38:31.005791 15257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:38:31.013283 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 15257
I20260923 14:38:31.013396 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:38:31.013652 2920 external_mini_cluster.cc:1428] Setting key a743e98344632994c1795405d52c2103
I20260923 14:38:31.015460 15387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:36341 every 8 connection(s)
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174310, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:31.043654 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:31.017894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:44599 (local address 127.2.218.62:36987)
0923 14:38:31.030227 (+ 12333us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:31.030235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:31.032113 (+ 1878us) server_negotiation.cc:407] Connection header received
0923 14:38:31.033184 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:31.033187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:31.033236 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:31.033289 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:31.034931 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.035637 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.037057 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.037351 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.037419 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:31.039966 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:31.039969 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:31.039971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:31.039993 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:31.041606 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:31.042148 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:31.042151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:31.042153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:31.042227 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:31.042557 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:31.042560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:31.042561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:31.042739 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:31.042823 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:31.043395 (+ 572us) server_negotiation.cc:299] Negotiation successful
0923 14:38:31.043504 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12263,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
I20260923 14:38:31.044945 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:31.017412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36987 (local address 127.2.218.1:44599)
0923 14:38:31.031982 (+ 14570us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:31.032029 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:31.032930 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:31.033800 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:31.033809 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:31.034157 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:31.034759 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:31.034776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.035775 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.035778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:31.036940 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:31.036949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.037594 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.037597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:31.037767 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:31.038382 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:31.038399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:31.039808 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:31.041727 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:31.041732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:31.041743 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:31.042026 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:31.042320 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:31.042323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:31.042325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:31.042465 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:31.042885 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:31.042890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:31.043152 (+ 262us) client_negotiation.cc:769] Sending connection context
0923 14:38:31.043818 (+ 666us) client_negotiation.cc:240] Negotiation successful
0923 14:38:31.044089 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14379,"thread_start_us":8672,"threads_started":1}
I20260923 14:38:31.046415 15388 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36987
I20260923 14:38:31.046622 15388 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:31.047094 15388 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
I20260923 14:38:31.048633 15174 ts_manager.cc:194] Registered new tserver with Master: 371f0e39086c4563b09e239daf4bee7a (127.2.218.1:36341)
I20260923 14:38:31.049907 15174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:44599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:38:31.071683 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:55795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36987
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 14:38:31.199599 15396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:31.199832 15396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:31.199936 15396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:31.203485 15396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:31.203578 15396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:31.203667 15396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:38:31.208405 15396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.218.20:55795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:31.209236 15396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:31.210048 15396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:31.219555 15405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:31.219870 15403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:31.220075 15402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:31.220234 15396 server_base.cc:1034] running on GCE node
I20260923 14:38:31.221329 15396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:31.221885 15396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:31.223061 15396 hybrid_clock.cc:648] HybridClock initialized: now 1790174311223037 us; error 45 us; skew 500 ppm
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:31.225641 15396 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:38:31.226576 15396 webserver.cc:460] Webserver started at http://127.2.218.2:46423/ using document root <none> and password file <none>
I20260923 14:38:31.227039 15396 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:31.227118 15396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:31.227332 15396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:31.240494 15396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "206301e9e26c4d5399fb75be1ec5790b"
format_stamp: "Formatted at 2026-09-23 14:38:31 on dist-test-slave-zl47"
server_key: "a225c9004a7beb16e2aa847421772245"
server_key_iv: "97592484a63dd4392a49a7628ecd6f05"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:31.241160 15396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "206301e9e26c4d5399fb75be1ec5790b"
format_stamp: "Formatted at 2026-09-23 14:38:31 on dist-test-slave-zl47"
server_key: "a225c9004a7beb16e2aa847421772245"
server_key_iv: "97592484a63dd4392a49a7628ecd6f05"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:31.245550 15396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260923 14:38:31.258397 15412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:31.259711 15396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:38:31.259805 15396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "206301e9e26c4d5399fb75be1ec5790b"
format_stamp: "Formatted at 2026-09-23 14:38:31 on dist-test-slave-zl47"
server_key: "a225c9004a7beb16e2aa847421772245"
server_key_iv: "97592484a63dd4392a49a7628ecd6f05"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:31.259891 15396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:31.285650 15396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:31.288312 15396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:31.288483 15396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:31.288995 15396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:31.289716 15396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:31.289775 15396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:31.289849 15396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:31.289891 15396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:31.299753 15396 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:45353
I20260923 14:38:31.299799 15525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:45353 every 8 connection(s)
I20260923 14:38:31.300532 15396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:38:31.302827 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 15396
I20260923 14:38:31.302944 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:38:31.303191 2920 external_mini_cluster.cc:1428] Setting key 880fe32a6051c13cc880ae5e0b5d086f
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
I20260923 14:38:31.318337 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:31.302254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:43187 (local address 127.2.218.62:36987)
0923 14:38:31.302360 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:31.302364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:31.304360 (+ 1996us) server_negotiation.cc:407] Connection header received
0923 14:38:31.307424 (+ 3064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:31.307429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:31.307483 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:31.307534 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:31.308943 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.309517 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.310828 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.311312 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.311406 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:31.314353 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:31.314358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:31.314362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:31.314393 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:31.315941 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:31.316407 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:31.316412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:31.316413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:31.316492 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:31.316856 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:31.316861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:31.316864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:31.317129 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:31.317222 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:31.317913 (+ 691us) server_negotiation.cc:299] Negotiation successful
0923 14:38:31.318059 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:38:31.319679 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:31.302468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36987 (local address 127.2.218.2:43187)
0923 14:38:31.303822 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:31.304285 (+ 463us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:31.307018 (+ 2733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:31.307886 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:31.307896 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:31.308192 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:31.308778 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:31.308808 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.309631 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.309634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:31.310677 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:31.310689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.311609 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.311613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:31.311834 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:31.312514 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:31.312552 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:31.314176 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:31.316050 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:31.316055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:31.316066 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:31.316278 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:31.316593 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:31.316598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:31.316603 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:31.316752 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:31.317307 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:31.317316 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:31.317627 (+ 311us) client_negotiation.cc:769] Sending connection context
0923 14:38:31.318475 (+ 848us) client_negotiation.cc:240] Negotiation successful
0923 14:38:31.318735 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"thread_start_us":119,"threads_started":1}
I20260923 14:38:31.320957 15526 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36987
I20260923 14:38:31.321159 15526 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:31.321578 15526 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
I20260923 14:38:31.322579 15174 ts_manager.cc:194] Registered new tserver with Master: 206301e9e26c4d5399fb75be1ec5790b (127.2.218.2:45353)
I20260923 14:38:31.323362 15174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:43187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:38:31.333091 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:55795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36987
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:31.454536 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:31.454743 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:31.454800 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:31.458809 15531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:31.458877 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:31.458971 15531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:38:31.463894 15531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.218.20:55795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:31.464937 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:31.465960 15531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:31.474901 15537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:31.474975 15536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:31.475144 15539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:31.475800 15531 server_base.cc:1034] running on GCE node
I20260923 14:38:31.476150 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:31.476676 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:31.477917 15531 hybrid_clock.cc:648] HybridClock initialized: now 1790174311477892 us; error 37 us; skew 500 ppm
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:31.480646 15531 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:38:31.481405 15531 webserver.cc:460] Webserver started at http://127.2.218.3:33241/ using document root <none> and password file <none>
I20260923 14:38:31.481845 15531 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:31.481921 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:31.482187 15531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:31.498224 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "ab0659400f3e4d00ad4e6bedeafe8212"
format_stamp: "Formatted at 2026-09-23 14:38:31 on dist-test-slave-zl47"
server_key: "99d2496bc69c695d6dd6655eee9b8830"
server_key_iv: "88aad1fa305c8384269e4cdabcdf1608"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:31.498860 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "ab0659400f3e4d00ad4e6bedeafe8212"
format_stamp: "Formatted at 2026-09-23 14:38:31 on dist-test-slave-zl47"
server_key: "99d2496bc69c695d6dd6655eee9b8830"
server_key_iv: "88aad1fa305c8384269e4cdabcdf1608"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:31.503212 15531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260923 14:38:31.518455 15546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:31.519717 15531 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260923 14:38:31.519811 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "ab0659400f3e4d00ad4e6bedeafe8212"
format_stamp: "Formatted at 2026-09-23 14:38:31 on dist-test-slave-zl47"
server_key: "99d2496bc69c695d6dd6655eee9b8830"
server_key_iv: "88aad1fa305c8384269e4cdabcdf1608"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:31.519913 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:31.532542 15531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:31.534874 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:31.535029 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:31.535488 15531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:31.536149 15531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:31.536221 15531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:31.536280 15531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:31.536338 15531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:31.544728 15531 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:43821
I20260923 14:38:31.545701 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:38:31.548885 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 15531
I20260923 14:38:31.549000 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:38:31.549233 2920 external_mini_cluster.cc:1428] Setting key b3f86341ecb6437747fc4f74c4b1a21a
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:31.556840 15659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:43821 every 8 connection(s)
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:31.577410 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:31.557416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:44627 (local address 127.2.218.62:36987)
0923 14:38:31.557534 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:31.557538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:31.558247 (+ 709us) server_negotiation.cc:407] Connection header received
0923 14:38:31.559259 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:31.559262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:31.559314 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:31.559387 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:31.560463 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.561161 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.569631 (+ 8470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.569963 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.570055 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:31.572704 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:31.572708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:31.572710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:31.572735 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:31.574421 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:31.575189 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:31.575192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:31.575194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:31.575276 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:31.576256 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:31.576259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:31.576261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:31.576470 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:31.576549 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:31.577160 (+ 611us) server_negotiation.cc:299] Negotiation successful
0923 14:38:31.577269 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:38:31.578926 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:31.557712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36987 (local address 127.2.218.3:44627)
0923 14:38:31.558133 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:31.558175 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:31.559053 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:31.559562 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:31.559568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:31.559850 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:31.560339 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:31.560351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.568277 (+ 7926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.568280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:31.569501 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:31.569522 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:31.570217 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:31.570221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:31.570412 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:31.571133 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:31.571150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:31.572551 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:31.574558 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:31.574563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:31.574576 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:31.574884 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:31.575994 (+ 1110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:31.575997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:31.575999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:31.576153 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:31.576611 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:31.576616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:31.576892 (+ 276us) client_negotiation.cc:769] Sending connection context
0923 14:38:31.577775 (+ 883us) client_negotiation.cc:240] Negotiation successful
0923 14:38:31.578046 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":130,"threads_started":1}
I20260923 14:38:31.580098 15660 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36987
I20260923 14:38:31.580331 15660 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:31.580891 15660 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
I20260923 14:38:31.581830 15174 ts_manager.cc:194] Registered new tserver with Master: ab0659400f3e4d00ad4e6bedeafe8212 (127.2.218.3:43821)
I20260923 14:38:31.582459 15174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:44627
I20260923 14:38:31.586453 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:38:31.587163 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 15257
I20260923 14:38:31.593215 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 15396
I20260923 14:38:31.601948 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 15531
I20260923 14:38:31.606894 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 15142
I20260923 14:38:31.612545 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36987
--webserver_interface=127.2.218.62
--webserver_port=39731
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:36987
--ranger_config_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:55795/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.218.20:55795/kms with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:31.704579 15663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:31.704766 15663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:31.704797 15663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:31.708269 15663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:38:31.708318 15663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:31.708336 15663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:31.708350 15663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:38:31.708364 15663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:38:31.712764 15663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.218.20:55795/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:36987
--ranger_config_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:31.713541 15663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:31.714381 15663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:31.721131 15669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:31.721187 15663 server_base.cc:1034] running on GCE node
W20260923 14:38:31.721122 15671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:31.721118 15668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:31.721897 15663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:31.722754 15663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:31.723942 15663 hybrid_clock.cc:648] HybridClock initialized: now 1790174311723940 us; error 40 us; skew 500 ppm
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:31.726303 15663 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:38:31.727047 15663 webserver.cc:460] Webserver started at http://127.2.218.62:39731/ using document root <none> and password file <none>
I20260923 14:38:31.727483 15663 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:31.727566 15663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:31.798544 15663 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:31.800364 15679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:31.801450 15663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:31.801555 15663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "353a68d0ee7c4d5a843bc4a0020b504c"
format_stamp: "Formatted at 2026-09-23 14:38:29 on dist-test-slave-zl47"
server_key: "82d180d059b6da102ea8b04701396d41"
server_key_iv: "15d20a529b965b6b37ef3bb48fbbfb3a"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:31.802058 15663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:31.809617 15663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:31.811682 15663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:31.811836 15663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:31.817703 15663 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:36987
I20260923 14:38:31.817734 15731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:36987 every 8 connection(s)
I20260923 14:38:31.818580 15663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:38:31.819020 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 15663
I20260923 14:38:31.819461 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:55795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:36341
--local_ip_for_outbound_sockets=127.2.218.1
--tserver_master_addrs=127.2.218.62:36987
--webserver_port=40273
--webserver_interface=127.2.218.1
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.218.20:55795/kms with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:38:31.822965 15732 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:38:31.824256 15732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: Bootstrap starting.
I20260923 14:38:31.839135 15732 log.cc:824] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:31.844519 15732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:38:31.844935 15732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: Bootstrap complete.
I20260923 14:38:31.848783 15732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } }
I20260923 14:38:31.849115 15732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353a68d0ee7c4d5a843bc4a0020b504c, State: Initialized, Role: FOLLOWER
I20260923 14:38:31.849552 15732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } }
I20260923 14:38:31.849656 15732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:38:31.849735 15732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:38:31.849848 15732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:38:31.851536 15732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } }
I20260923 14:38:31.851841 15732 leader_election.cc:302] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 353a68d0ee7c4d5a843bc4a0020b504c; no voters:
I20260923 14:38:31.852097 15732 leader_election.cc:288] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 14:38:31.852216 15736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 2 FOLLOWER]: Leader election won for term 2
I20260923 14:38:31.852432 15736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [term 2 LEADER]: Becoming Leader. State: Replica: 353a68d0ee7c4d5a843bc4a0020b504c, State: Running, Role: LEADER
I20260923 14:38:31.852751 15736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } }
I20260923 14:38:31.853121 15732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:38:31.853904 15737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } } }
I20260923 14:38:31.854007 15737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:31.854748 15738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353a68d0ee7c4d5a843bc4a0020b504c. Latest consensus state: current_term: 2 leader_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353a68d0ee7c4d5a843bc4a0020b504c" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36987 } } }
I20260923 14:38:31.854832 15738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:31.855037 15745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:38:31.855127 15732 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 14:38:31.860141 15745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:38:31.860845 15745 catalog_manager.cc:1259] Loaded cluster ID: 341767961fc6479ea56c4c98c76873d1
I20260923 14:38:31.860895 15745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:38:31.863385 15745 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:38:31.864048 15745 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: Loaded TSK: 0
I20260923 14:38:31.865178 15745 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:38:31.944411 15734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:31.944660 15734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:31.944711 15734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:31.949699 15734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:31.949783 15734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:31.949874 15734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:38:31.956197 15734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.218.20:55795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:36341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=40273
--tserver_master_addrs=127.2.218.62:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:31.957155 15734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:31.958238 15734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:31.968258 15770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:31.968261 15769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:31.968659 15734 server_base.cc:1034] running on GCE node
W20260923 14:38:31.968330 15772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:31.969149 15734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:31.969822 15734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:31.971007 15734 hybrid_clock.cc:648] HybridClock initialized: now 1790174311970990 us; error 29 us; skew 500 ppm
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:31.974020 15734 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:38:31.974975 15734 webserver.cc:460] Webserver started at http://127.2.218.1:40273/ using document root <none> and password file <none>
I20260923 14:38:31.975572 15734 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:31.975688 15734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 14:38:31 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:32.002753 15734 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260923 14:38:32.007081 15780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:32.008541 15734 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 14:38:32.008648 15734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "371f0e39086c4563b09e239daf4bee7a"
format_stamp: "Formatted at 2026-09-23 14:38:30 on dist-test-slave-zl47"
server_key: "8d69c3a96e4903beeb537e2fff060b29"
server_key_iv: "0069b9cfb0983e8fcde0360a4cfd11aa"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:32.009266 15734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:32.030603 15734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:32.033975 15734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:32.034160 15734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:32.034734 15734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:32.035665 15734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:32.035725 15734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:32.035776 15734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:32.035830 15734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:32.047518 15734 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:36341
I20260923 14:38:32.048607 15734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:38:32.050199 15893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:36341 every 8 connection(s)
I20260923 14:38:32.056027 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 15734
I20260923 14:38:32.056527 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:55795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:45353
--local_ip_for_outbound_sockets=127.2.218.2
--tserver_master_addrs=127.2.218.62:36987
--webserver_port=46423
--webserver_interface=127.2.218.2
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.218.20:55795/kms with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:38:32 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174311, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:32.069259 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:32.051706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36987 (local address 127.2.218.1:49523)
0923 14:38:32.054678 (+ 2972us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:32.054715 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:32.055534 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:32.056565 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:32.056571 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:32.056989 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:32.057368 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:32.057380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.059049 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.059053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:32.060229 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:32.060241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.061280 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.061287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:32.061484 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:32.062114 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:32.062136 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:32.064037 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:32.066482 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:32.066487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:32.066499 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:32.066817 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:32.067164 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:32.067167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:32.067169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:32.067305 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:32.067880 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:32.067886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:32.068112 (+ 226us) client_negotiation.cc:769] Sending connection context
0923 14:38:32.068246 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 14:38:32.068486 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2811,"thread_start_us":118,"threads_started":1}
I20260923 14:38:32.071161 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:32.051610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:49523 (local address 127.2.218.62:36987)
0923 14:38:32.052162 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:32.052183 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:32.054840 (+ 2657us) server_negotiation.cc:407] Connection header received
0923 14:38:32.055919 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:32.055928 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:32.056233 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:32.056507 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:32.057519 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.058844 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.060359 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.061103 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.061836 (+ 733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:32.064239 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:32.064248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:32.064262 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:32.064295 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:32.066323 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:32.066937 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:32.066942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:32.066947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:32.067062 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:32.067404 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:32.067406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:32.067408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:32.067757 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:32.070025 (+ 2268us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:32.070216 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 14:38:32.070434 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":128,"threads_started":1}
I20260923 14:38:32.073397 15894 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36987
I20260923 14:38:32.073575 15894 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:32.073992 15894 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
W20260923 14:38:32.075953 15894 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:36987 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 14:38:32.076074 15894 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:32.076228 15894 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
I20260923 14:38:32.077033 15894 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:32.077201 15894 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
W20260923 14:38:32.077972 15894 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:38:32.172024 15898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:32.172310 15898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:32.172369 15898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:32.178016 15898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:32.178082 15898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:32.178195 15898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:38:32.184283 15898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.218.20:55795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:45353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=46423
--tserver_master_addrs=127.2.218.62:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:32.185230 15898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:32.186087 15898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:32.195582 15903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:32.195560 15906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:32.197767 15904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:32.197942 15898 server_base.cc:1034] running on GCE node
I20260923 14:38:32.198302 15898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:32.198958 15898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:32.200186 15898 hybrid_clock.cc:648] HybridClock initialized: now 1790174312200172 us; error 36 us; skew 500 ppm
Sep 23 14:38:32 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:32.203303 15898 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:38:32.204113 15898 webserver.cc:460] Webserver started at http://127.2.218.2:46423/ using document root <none> and password file <none>
I20260923 14:38:32.204674 15898 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:32.204767 15898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 14:38:32 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:32.222569 15898 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:38:32.225268 15914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:32.226570 15898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:38:32.226702 15898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "206301e9e26c4d5399fb75be1ec5790b"
format_stamp: "Formatted at 2026-09-23 14:38:31 on dist-test-slave-zl47"
server_key: "a225c9004a7beb16e2aa847421772245"
server_key_iv: "97592484a63dd4392a49a7628ecd6f05"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:32.227170 15898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:32.245010 15898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:32.248162 15898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:32.248387 15898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:32.248981 15898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:32.249877 15898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:32.249975 15898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:32.250068 15898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:32.250152 15898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:32.262784 15898 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:45353
I20260923 14:38:32.262830 16027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:45353 every 8 connection(s)
I20260923 14:38:32.263747 15898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:38:32.264919 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 15898
I20260923 14:38:32.265360 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.218.20:55795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:43821
--local_ip_for_outbound_sockets=127.2.218.3
--tserver_master_addrs=127.2.218.62:36987
--webserver_port=33241
--webserver_interface=127.2.218.3
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.218.20:55795/kms with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:38:32 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:32.281060 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:32.268971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:58107 (local address 127.2.218.62:36987)
0923 14:38:32.269087 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:32.269091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:32.269107 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:38:32.269584 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:32.269588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:32.269645 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:32.269719 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:32.270984 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.271785 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.273576 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.273949 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.274043 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:32.276752 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:32.276757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:32.276762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:32.276797 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:32.278670 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:32.279309 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:32.279315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:32.279319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:32.279411 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:32.279842 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:32.279847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:32.279849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:32.280084 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:32.280168 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:32.280766 (+ 598us) server_negotiation.cc:299] Negotiation successful
0923 14:38:32.280885 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:38:32.282517 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:32.267920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36987 (local address 127.2.218.2:58107)
0923 14:38:32.268478 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:32.268526 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:32.269390 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:32.269898 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:32.269904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:32.270203 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:32.270806 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:32.270828 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.271980 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.271985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:32.273420 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:32.273434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.274268 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.274286 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:32.274498 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:32.275094 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:32.275118 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:32.276555 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:32.278814 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:32.278822 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:32.278835 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:32.279166 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:32.279538 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:32.279542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:32.279546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:32.279718 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:32.280252 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:32.280257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:32.280522 (+ 265us) client_negotiation.cc:769] Sending connection context
0923 14:38:32.281435 (+ 913us) client_negotiation.cc:240] Negotiation successful
0923 14:38:32.281695 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":190,"threads_started":1}
I20260923 14:38:32.283762 16028 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36987
I20260923 14:38:32.283951 16028 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:32.284392 16028 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
W20260923 14:38:32.285903 16028 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:36987 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 14:38:32.286002 16028 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:32.286222 16028 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
I20260923 14:38:32.287030 16028 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:32.287176 16028 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
W20260923 14:38:32.287855 16028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:38:32.385967 16030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:32.386258 16030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:32.386325 16030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:32.389721 16030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:32.389791 16030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:32.389878 16030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:38:32.395354 16030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.218.20:55795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:43821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=33241
--tserver_master_addrs=127.2.218.62:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:32.396479 16030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:32.397523 16030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:32.407716 16038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:32.407902 16040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:32.408535 16037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:32.408656 16030 server_base.cc:1034] running on GCE node
I20260923 14:38:32.409119 16030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:32.409736 16030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:32.410914 16030 hybrid_clock.cc:648] HybridClock initialized: now 1790174312410897 us; error 31 us; skew 500 ppm
Sep 23 14:38:32 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:32.413906 16030 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:38:32.414809 16030 webserver.cc:460] Webserver started at http://127.2.218.3:33241/ using document root <none> and password file <none>
I20260923 14:38:32.415417 16030 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:32.415511 16030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 14:38:32 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:32.433663 16030 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:38:32.446400 16048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:32.447765 16030 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260923 14:38:32.447892 16030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "ab0659400f3e4d00ad4e6bedeafe8212"
format_stamp: "Formatted at 2026-09-23 14:38:31 on dist-test-slave-zl47"
server_key: "99d2496bc69c695d6dd6655eee9b8830"
server_key_iv: "88aad1fa305c8384269e4cdabcdf1608"
server_key_version: "kuduclusterkey@0"
I20260923 14:38:32.448505 16030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:32.458838 16030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:32.461423 16030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:32.461596 16030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:32.462217 16030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:32.463059 16030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:32.463140 16030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:32.463202 16030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:32.463249 16030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:32.473867 16030 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:43821
I20260923 14:38:32.475033 16030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:38:32.476025 16161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:43821 every 8 connection(s)
I20260923 14:38:32.476204 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 16030
Sep 23 14:38:32 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:32.490298 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:32.478060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.2.218.62:36987)
0923 14:38:32.479234 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:32.479237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:32.479246 (+ 9us) server_negotiation.cc:407] Connection header received
0923 14:38:32.479281 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:32.479283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:32.479325 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:32.479416 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:32.480225 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.480937 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.481994 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.482363 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.482477 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:32.483962 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:32.483967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:32.483970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:32.483997 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:32.485872 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:32.486365 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:32.486369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:32.486370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:32.486448 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:32.486729 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:32.486735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:32.486739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:32.486951 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:32.487150 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:32.489906 (+ 2756us) server_negotiation.cc:299] Negotiation successful
0923 14:38:32.490049 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1120,"thread_start_us":136,"threads_started":1}
I20260923 14:38:32.490738 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:32.477793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:54365 (local address 127.2.218.62:36987)
0923 14:38:32.477944 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:32.477949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:32.479881 (+ 1932us) server_negotiation.cc:407] Connection header received
0923 14:38:32.480691 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:32.480693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:32.480736 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:32.480808 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:32.481898 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.482408 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.483924 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.484126 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.484175 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:32.486986 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:32.486989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:32.486991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:32.487014 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:32.488463 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:32.489004 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:32.489007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:32.489008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:32.489086 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:32.489435 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:32.489438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:32.489440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:32.489584 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:32.489690 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:32.490511 (+ 821us) server_negotiation.cc:299] Negotiation successful
0923 14:38:32.490610 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 14:38:32.492275 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:32.478256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:36987 (local address 127.2.218.3:54365)
0923 14:38:32.479710 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:32.479759 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:32.480524 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:32.480944 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:32.480950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:32.481237 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:32.481755 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:32.481771 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.482558 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.482562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:32.483782 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:32.483790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:32.484389 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:32.484394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:32.484579 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:32.485085 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:32.485108 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:32.486838 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:32.488583 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:32.488587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:32.488599 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:32.488901 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:32.489192 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:32.489196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:32.489198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:32.489345 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:32.489692 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:32.489698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:32.490003 (+ 305us) client_negotiation.cc:769] Sending connection context
0923 14:38:32.490920 (+ 917us) client_negotiation.cc:240] Negotiation successful
0923 14:38:32.491196 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1263,"thread_start_us":143,"threads_started":1}
I20260923 14:38:32.493309 16162 heartbeater.cc:347] Connected to a master server at 127.2.218.62:36987
I20260923 14:38:32.493506 16162 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:32.493978 16162 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
W20260923 14:38:32.495476 16162 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:36987 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 14:38:32.495579 16162 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:32.495715 16162 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
I20260923 14:38:32.496369 16162 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:32.496539 16162 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
W20260923 14:38:32.497195 16162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 14:38:32 dist-test-slave-zl47 krb5kdc[14699](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790174312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:33.078219 15894 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:33.078511 15894 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
I20260923 14:38:33.079704 15696 ts_manager.cc:194] Registered new tserver with Master: 371f0e39086c4563b09e239daf4bee7a (127.2.218.1:36341)
I20260923 14:38:33.081179 15696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:49523
Sep 23 14:38:33 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:33.288058 16028 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:33.288340 16028 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
I20260923 14:38:33.289103 15696 ts_manager.cc:194] Registered new tserver with Master: 206301e9e26c4d5399fb75be1ec5790b (127.2.218.2:45353)
I20260923 14:38:33.289927 15696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:58107
Sep 23 14:38:33 dist-test-slave-zl47 krb5kdc[14699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174312, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for HTTP/127.2.218.20@KRBTEST.COM
I20260923 14:38:33.497394 16162 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:33.497641 16162 heartbeater.cc:511] Master 127.2.218.62:36987 requested a full tablet report, sending...
I20260923 14:38:33.498322 15696 ts_manager.cc:194] Registered new tserver with Master: ab0659400f3e4d00ad4e6bedeafe8212 (127.2.218.3:43821)
I20260923 14:38:33.499034 15696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:54365
I20260923 14:38:33.504354 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:38:33.522310 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.510442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.2.218.62:36987)
0923 14:38:33.513627 (+ 3185us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:33.513631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:33.513642 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:38:33.513675 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:33.513677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:33.513717 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:33.513799 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:33.514654 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.515381 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.516529 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.516845 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.516933 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.518725 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:33.518728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:33.518731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:33.518753 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:33.520420 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:33.520840 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:33.520843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:33.520845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:33.520918 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:33.521185 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:33.521188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:33.521190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:33.521361 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:33.521464 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:33.521999 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 14:38:33.522157 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3119,"thread_start_us":74,"threads_started":1}
I20260923 14:38:33.525929 15696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:38:33.530308 15696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:38:33.553884 16195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.542864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:45353 (local address 127.0.0.1:38092)
0923 14:38:33.547821 (+ 4957us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:33.547833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:33.547908 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:33.548492 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:33.548497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:33.548516 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:33.548620 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:33.548625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.550518 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.550521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:33.551678 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:33.551687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.553334 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.553337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:33.553428 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.553463 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 14:38:33.553618 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 14:38:33.553759 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4899,"thread_start_us":69,"threads_started":1}
I20260923 14:38:33.555709 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.542983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.2.218.2:45353)
0923 14:38:33.547698 (+ 4715us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:33.547704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:33.547986 (+ 282us) server_negotiation.cc:407] Connection header received
0923 14:38:33.548054 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:33.548059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:33.548228 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:33.548346 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:33.548733 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.549766 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.552040 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.553064 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.553154 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.553272 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:33.555496 (+ 2224us) server_negotiation.cc:299] Negotiation successful
0923 14:38:33.555587 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4624,"thread_start_us":83,"threads_started":1}
I20260923 14:38:33.558010 16193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.541359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:36341 (local address 127.0.0.1:48580)
0923 14:38:33.543942 (+ 2583us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:33.543984 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:33.544093 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:33.551836 (+ 7743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:33.551839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:33.551856 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:33.551931 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:33.551936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.555484 (+ 3548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.555487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:33.556598 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:33.556609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.557672 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.557676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:33.557759 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.557782 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 14:38:33.557832 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 14:38:33.557887 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2415,"thread_start_us":74,"threads_started":1}
I20260923 14:38:33.558826 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.541499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.2.218.1:36341)
0923 14:38:33.550705 (+ 9206us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:33.550710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:33.550731 (+ 21us) server_negotiation.cc:407] Connection header received
0923 14:38:33.550803 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:33.550807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:33.550956 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:33.551083 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:33.552649 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.555348 (+ 2699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.556749 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.557537 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.558443 (+ 906us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.558566 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:33.558612 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 14:38:33.558688 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9115,"thread_start_us":85,"threads_started":1}
I20260923 14:38:33.558694 15962 tablet_service.cc:1467] Processing CreateTablet for tablet dff1d9f53f6144a7a13f78dbdf1a0727 (DEFAULT_TABLE table=test-table [id=788f0a03f1d34361a31a2625c0984df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:33.559720 15962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff1d9f53f6144a7a13f78dbdf1a0727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:33.561380 15828 tablet_service.cc:1467] Processing CreateTablet for tablet dff1d9f53f6144a7a13f78dbdf1a0727 (DEFAULT_TABLE table=test-table [id=788f0a03f1d34361a31a2625c0984df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:33.562285 15828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff1d9f53f6144a7a13f78dbdf1a0727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:33.572757 16197 tablet_bootstrap.cc:492] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b: Bootstrap starting.
I20260923 14:38:33.573100 16198 tablet_bootstrap.cc:492] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a: Bootstrap starting.
I20260923 14:38:33.573395 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.542321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:43821 (local address 127.0.0.1:45372)
0923 14:38:33.544283 (+ 1962us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:33.544294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:33.544357 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:33.568805 (+ 24448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:33.568808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:33.568839 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:33.568923 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:33.568928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.570447 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.570450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:33.571463 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:33.571472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.572393 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.572397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:33.572489 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.572510 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 14:38:33.572578 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 14:38:33.572632 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1911,"thread_start_us":80,"threads_started":1}
I20260923 14:38:33.574038 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.542467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.2.218.3:43821)
0923 14:38:33.566553 (+ 24086us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:33.566559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:33.566579 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:38:33.566643 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:33.566647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:33.566797 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:33.569056 (+ 2259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:33.569189 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.570305 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.571600 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.572269 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.572369 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.572481 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:33.573810 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0923 14:38:33.573903 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23992,"thread_start_us":71,"threads_started":1}
I20260923 14:38:33.575280 16197 tablet_bootstrap.cc:654] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:33.576004 16197 log.cc:824] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:33.576512 16081 tablet_service.cc:1467] Processing CreateTablet for tablet dff1d9f53f6144a7a13f78dbdf1a0727 (DEFAULT_TABLE table=test-table [id=788f0a03f1d34361a31a2625c0984df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:33.577360 16081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff1d9f53f6144a7a13f78dbdf1a0727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:33.577772 16197 tablet_bootstrap.cc:492] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b: No bootstrap required, opened a new log
I20260923 14:38:33.577924 16197 ts_tablet_manager.cc:1397] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:38:33.577996 16198 tablet_bootstrap.cc:654] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:33.578656 16198 log.cc:824] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:33.580264 16198 tablet_bootstrap.cc:492] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a: No bootstrap required, opened a new log
I20260923 14:38:33.580415 16198 ts_tablet_manager.cc:1397] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260923 14:38:33.583905 16198 raft_consensus.cc:348] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371f0e39086c4563b09e239daf4bee7a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36341 } } peers { permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 } } peers { permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 } }
I20260923 14:38:33.584142 16198 raft_consensus.cc:374] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:33.584205 16198 raft_consensus.cc:729] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371f0e39086c4563b09e239daf4bee7a, State: Initialized, Role: FOLLOWER
I20260923 14:38:33.584617 16198 consensus_queue.cc:260] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371f0e39086c4563b09e239daf4bee7a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36341 } } peers { permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 } } peers { permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 } }
I20260923 14:38:33.585536 16198 ts_tablet_manager.cc:1428] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 14:38:33.585863 15894 heartbeater.cc:503] Master 127.2.218.62:36987 was elected leader, sending a full tablet report...
I20260923 14:38:33.587086 16201 tablet_bootstrap.cc:492] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: Bootstrap starting.
I20260923 14:38:33.588739 16201 tablet_bootstrap.cc:654] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:33.589388 16201 log.cc:824] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:33.591606 16201 tablet_bootstrap.cc:492] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: No bootstrap required, opened a new log
I20260923 14:38:33.591786 16201 ts_tablet_manager.cc:1397] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:38:33.595367 16197 raft_consensus.cc:348] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371f0e39086c4563b09e239daf4bee7a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36341 } } peers { permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 } } peers { permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 } }
I20260923 14:38:33.595639 16197 raft_consensus.cc:374] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:33.595697 16197 raft_consensus.cc:729] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206301e9e26c4d5399fb75be1ec5790b, State: Initialized, Role: FOLLOWER
I20260923 14:38:33.596087 16197 consensus_queue.cc:260] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371f0e39086c4563b09e239daf4bee7a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36341 } } peers { permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 } } peers { permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 } }
I20260923 14:38:33.596028 16201 raft_consensus.cc:348] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371f0e39086c4563b09e239daf4bee7a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36341 } } peers { permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 } } peers { permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 } }
I20260923 14:38:33.596336 16201 raft_consensus.cc:374] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:33.596428 16201 raft_consensus.cc:729] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0659400f3e4d00ad4e6bedeafe8212, State: Initialized, Role: FOLLOWER
I20260923 14:38:33.596877 16197 ts_tablet_manager.cc:1428] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260923 14:38:33.596835 16201 consensus_queue.cc:260] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371f0e39086c4563b09e239daf4bee7a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36341 } } peers { permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 } } peers { permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 } }
I20260923 14:38:33.597150 16028 heartbeater.cc:503] Master 127.2.218.62:36987 was elected leader, sending a full tablet report...
I20260923 14:38:33.597741 16201 ts_tablet_manager.cc:1428] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:38:33.599190 16162 heartbeater.cc:503] Master 127.2.218.62:36987 was elected leader, sending a full tablet report...
I20260923 14:38:33.652209 16202 raft_consensus.cc:482] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:38:33.652415 16202 raft_consensus.cc:504] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371f0e39086c4563b09e239daf4bee7a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36341 } } peers { permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 } } peers { permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 } }
I20260923 14:38:33.653693 16202 leader_election.cc:288] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab0659400f3e4d00ad4e6bedeafe8212 (127.2.218.3:43821), 206301e9e26c4d5399fb75be1ec5790b (127.2.218.2:45353)
I20260923 14:38:33.658010 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.654178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:46537 (local address 127.2.218.3:43821)
0923 14:38:33.654286 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:33.654291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:33.654470 (+ 179us) server_negotiation.cc:407] Connection header received
0923 14:38:33.654550 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:33.654552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:33.654596 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:33.654677 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:33.655041 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.655859 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.656912 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.657444 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.657569 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.657636 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:33.657841 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 14:38:33.657902 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:38:33.658022 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.653877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:43821 (local address 127.2.218.1:46537)
0923 14:38:33.654392 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:33.654401 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:33.654475 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:33.654703 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:33.654706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:33.654722 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:33.654933 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:33.654938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.655969 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.655972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:33.656738 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:33.656746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.657600 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.657603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:33.657713 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.657737 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 14:38:33.657828 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 14:38:33.657886 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":197,"thread_start_us":113,"threads_started":1}
I20260923 14:38:33.658610 16101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff1d9f53f6144a7a13f78dbdf1a0727" candidate_uuid: "371f0e39086c4563b09e239daf4bee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" is_pre_election: true
I20260923 14:38:33.658876 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.654085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:37237 (local address 127.2.218.2:45353)
0923 14:38:33.654330 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:33.654334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:33.654398 (+ 64us) server_negotiation.cc:407] Connection header received
0923 14:38:33.654551 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:33.654554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:33.654602 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:33.654636 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:33.656019 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.656725 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.657708 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.658263 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.658365 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.658413 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:33.658705 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 14:38:33.658779 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260923 14:38:33.658878 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.653877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:45353 (local address 127.2.218.1:37237)
0923 14:38:33.654296 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:33.654310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:33.654424 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:33.654777 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:33.654779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:33.654789 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:33.654933 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:33.654938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.656853 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.656857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:33.657603 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:33.657611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.658467 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.658470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:33.658556 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.658576 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:38:33.658653 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 14:38:33.658739 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":7,"thread_start_us":103,"threads_started":1}
I20260923 14:38:33.658926 16101 raft_consensus.cc:2454] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371f0e39086c4563b09e239daf4bee7a in term 0.
I20260923 14:38:33.659391 15782 leader_election.cc:302] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371f0e39086c4563b09e239daf4bee7a, ab0659400f3e4d00ad4e6bedeafe8212; no voters:
I20260923 14:38:33.659642 15969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff1d9f53f6144a7a13f78dbdf1a0727" candidate_uuid: "371f0e39086c4563b09e239daf4bee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206301e9e26c4d5399fb75be1ec5790b" is_pre_election: true
I20260923 14:38:33.659888 15969 raft_consensus.cc:2454] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 371f0e39086c4563b09e239daf4bee7a in term 0.
I20260923 14:38:33.660144 16202 raft_consensus.cc:2781] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:38:33.660219 16202 raft_consensus.cc:482] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:38:33.660254 16202 raft_consensus.cc:3037] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:33.661468 16202 raft_consensus.cc:504] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371f0e39086c4563b09e239daf4bee7a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36341 } } peers { permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 } } peers { permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 } }
I20260923 14:38:33.661789 16202 leader_election.cc:288] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [CANDIDATE]: Term 1 election: Requested vote from peers ab0659400f3e4d00ad4e6bedeafe8212 (127.2.218.3:43821), 206301e9e26c4d5399fb75be1ec5790b (127.2.218.2:45353)
I20260923 14:38:33.662210 15969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff1d9f53f6144a7a13f78dbdf1a0727" candidate_uuid: "371f0e39086c4563b09e239daf4bee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206301e9e26c4d5399fb75be1ec5790b"
I20260923 14:38:33.662242 16101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff1d9f53f6144a7a13f78dbdf1a0727" candidate_uuid: "371f0e39086c4563b09e239daf4bee7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0659400f3e4d00ad4e6bedeafe8212"
I20260923 14:38:33.662348 16101 raft_consensus.cc:3037] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:33.662352 15969 raft_consensus.cc:3037] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:33.663390 15969 raft_consensus.cc:2454] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371f0e39086c4563b09e239daf4bee7a in term 1.
I20260923 14:38:33.663522 16101 raft_consensus.cc:2454] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 371f0e39086c4563b09e239daf4bee7a in term 1.
I20260923 14:38:33.663763 15782 leader_election.cc:302] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 371f0e39086c4563b09e239daf4bee7a, ab0659400f3e4d00ad4e6bedeafe8212; no voters:
I20260923 14:38:33.663929 16202 raft_consensus.cc:2781] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:38:33.664104 16202 raft_consensus.cc:686] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [term 1 LEADER]: Becoming Leader. State: Replica: 371f0e39086c4563b09e239daf4bee7a, State: Running, Role: LEADER
I20260923 14:38:33.664383 16202 consensus_queue.cc:237] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371f0e39086c4563b09e239daf4bee7a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36341 } } peers { permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 } } peers { permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 } }
I20260923 14:38:33.667901 15696 catalog_manager.cc:5495] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 371f0e39086c4563b09e239daf4bee7a (127.2.218.1). New cstate: current_term: 1 leader_uuid: "371f0e39086c4563b09e239daf4bee7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "371f0e39086c4563b09e239daf4bee7a" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 } health_report { overall_health: UNKNOWN } } }
I20260923 14:38:33.701200 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.697786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48596 (local address 127.2.218.1:36341)
0923 14:38:33.697909 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:33.697914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:33.697927 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:38:33.698066 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:33.698069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:33.698140 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:33.698234 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:38:33.698458 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.699118 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.699871 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:33.700065 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:33.700173 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:33.700462 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:38:33.700805 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:38:33.700928 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:33.700986 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 14:38:33.701050 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:38:33.706642 16101 raft_consensus.cc:1261] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212 [term 1 FOLLOWER]: Refusing update from remote peer 371f0e39086c4563b09e239daf4bee7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:38:33.707213 16202 consensus_queue.cc:1035] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0659400f3e4d00ad4e6bedeafe8212" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:38:33.707427 15969 raft_consensus.cc:1261] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b [term 1 FOLLOWER]: Refusing update from remote peer 371f0e39086c4563b09e239daf4bee7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:38:33.707896 16202 consensus_queue.cc:1035] T dff1d9f53f6144a7a13f78dbdf1a0727 P 371f0e39086c4563b09e239daf4bee7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "206301e9e26c4d5399fb75be1ec5790b" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 45353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:38:33.715397 16218 mvcc.cc:205] Tried to move back new op lower bound from 7332553988935229440 to 7332553988772278272. Current Snapshot: MvccSnapshot[applied={T|T < 7332553988935229440}]
I20260923 14:38:33.715770 16220 mvcc.cc:205] Tried to move back new op lower bound from 7332553988935229440 to 7332553988772278272. Current Snapshot: MvccSnapshot[applied={T|T < 7332553988935229440 or (T in {7332553988935229440})}]
I20260923 14:38:33.717301 16219 mvcc.cc:205] Tried to move back new op lower bound from 7332553988935229440 to 7332553988772278272. Current Snapshot: MvccSnapshot[applied={T|T < 7332553988935229440}]
I20260923 14:38:33.723039 15695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:38:33.723186 15695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:38:33.725216 15695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353a68d0ee7c4d5a843bc4a0020b504c: Sending DeleteTablet for 3 replicas of tablet dff1d9f53f6144a7a13f78dbdf1a0727
I20260923 14:38:33.726038 15962 tablet_service.cc:1514] Processing DeleteTablet for tablet dff1d9f53f6144a7a13f78dbdf1a0727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:33 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:38092
I20260923 14:38:33.726047 15828 tablet_service.cc:1514] Processing DeleteTablet for tablet dff1d9f53f6144a7a13f78dbdf1a0727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:33 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:48580
I20260923 14:38:33.726468 16227 tablet_replica.cc:330] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b: stopping tablet replica
I20260923 14:38:33.726492 16081 tablet_service.cc:1514] Processing DeleteTablet for tablet dff1d9f53f6144a7a13f78dbdf1a0727 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:33 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:45372
I20260923 14:38:33.726804 16227 raft_consensus.cc:2229] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:38:33.727083 16227 raft_consensus.cc:2258] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:38:33.727694 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 15734
I20260923 14:38:33.728621 16229 tablet_replica.cc:330] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: stopping tablet replica
I20260923 14:38:33.729250 16229 raft_consensus.cc:2229] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:38:33.729574 16229 raft_consensus.cc:2258] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:38:33.730777 16229 ts_tablet_manager.cc:1905] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:38:33.733095 16229 ts_tablet_manager.cc:1918] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:38:33.733197 16229 log.cc:1196] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/dff1d9f53f6144a7a13f78dbdf1a0727
I20260923 14:38:33.733544 16229 ts_tablet_manager.cc:1939] T dff1d9f53f6144a7a13f78dbdf1a0727 P ab0659400f3e4d00ad4e6bedeafe8212: Deleting consensus metadata
I20260923 14:38:33.734913 15681 catalog_manager.cc:4841] TS ab0659400f3e4d00ad4e6bedeafe8212 (127.2.218.3:43821): tablet dff1d9f53f6144a7a13f78dbdf1a0727 (table test-table [id=788f0a03f1d34361a31a2625c0984df0]) successfully deleted
I20260923 14:38:33.735138 16227 ts_tablet_manager.cc:1905] T dff1d9f53f6144a7a13f78dbdf1a0727 P 206301e9e26c4d5399fb75be1ec5790b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 14:38:33.735350 15681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:36341 (error 108)
I20260923 14:38:33.735538 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 15898
I20260923 14:38:33.736830 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.736590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:36341 (local address 127.0.0.1:48604)
0923 14:38:33.736701 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:33.736761 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:36341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 14:38:33.736994 15681 catalog_manager.cc:4584] TS 371f0e39086c4563b09e239daf4bee7a (127.2.218.1:36341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dff1d9f53f6144a7a13f78dbdf1a0727: Network error: Client connection negotiation failed: client connection to 127.2.218.1:36341: connect: Connection refused (error 111)
I20260923 14:38:33.742255 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 16030
I20260923 14:38:33.743237 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:33.742373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:45353 (local address 127.0.0.1:38102)
0923 14:38:33.743134 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:33.743179 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.2:45353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698}
I20260923 14:38:33.747241 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 15663
I20260923 14:38:33.753019 2920 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 14:38:34.097227 2920 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 14:38:34.097352 2920 mini_ranger.cc:66] Stopping Ranger...
I20260923 14:38:34.431383 2920 mini_ranger.cc:68] Stopped Ranger
2026-09-23 14:38:34.431 UTC [14729] LOG: received smart shutdown request
2026-09-23 14:38:34.435 UTC [14729] LOG: background worker "logical replication launcher" (PID 14738) exited with exit code 1
2026-09-23 14:38:34.435 UTC [14733] LOG: shutting down
2026-09-23 14:38:34.563 UTC [14729] LOG: database system is shut down
2026-09-23T14:38:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (42002 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:38:34 dist-test-slave-zl47 krb5kdc[16239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:38:34 dist-test-slave-zl47 krb5kdc[16239](info): set up 2 sockets
Sep 23 14:38:34 dist-test-slave-zl47 krb5kdc[16239](info): commencing operation
krb5kdc: starting...
W20260923 14:38:35.602214 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.904s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:38:35 dist-test-slave-zl47 krb5kdc[16239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:38:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:38:35.704770 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:34051
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:34051,127.2.218.61:34915,127.2.218.60:39351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:35.798764 16252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:35.798959 16252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:35.799036 16252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:35.802510 16252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:38:35.802554 16252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:35.802570 16252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:35.802623 16252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:38:35.802646 16252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:38:35.807164 16252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:34051,127.2.218.61:34915,127.2.218.60:39351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:35.808033 16252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:35.808882 16252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:35.816255 16258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:35.816265 16257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:35.816269 16260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:35.816398 16252 server_base.cc:1034] running on GCE node
I20260923 14:38:35.816893 16252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:35.817844 16252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:35.819012 16252 hybrid_clock.cc:648] HybridClock initialized: now 1790174315818974 us; error 53 us; skew 500 ppm
Sep 23 14:38:35 dist-test-slave-zl47 krb5kdc[16239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:35.824532 16252 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:38:35.825302 16252 webserver.cc:460] Webserver started at http://127.2.218.62:38431/ using document root <none> and password file <none>
I20260923 14:38:35.825721 16252 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:35.825771 16252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:35.825905 16252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:35.827580 16252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef"
format_stamp: "Formatted at 2026-09-23 14:38:35 on dist-test-slave-zl47"
server_key: "7225a4f5a39647b516db2d83d126423d"
server_key_iv: "58592093adbf4b8d9acc6797b83e9ea5"
server_key_version: "encryptionkey@0"
I20260923 14:38:35.828002 16252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef"
format_stamp: "Formatted at 2026-09-23 14:38:35 on dist-test-slave-zl47"
server_key: "7225a4f5a39647b516db2d83d126423d"
server_key_iv: "58592093adbf4b8d9acc6797b83e9ea5"
server_key_version: "encryptionkey@0"
I20260923 14:38:35.830928 16252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:38:35.832785 16267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:35.833742 16252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:35.833833 16252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef"
format_stamp: "Formatted at 2026-09-23 14:38:35 on dist-test-slave-zl47"
server_key: "7225a4f5a39647b516db2d83d126423d"
server_key_iv: "58592093adbf4b8d9acc6797b83e9ea5"
server_key_version: "encryptionkey@0"
I20260923 14:38:35.833900 16252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:35.853008 16252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:35.855212 16252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:35.855331 16252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:35.861449 16252 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:34051
I20260923 14:38:35.861472 16319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:34051 every 8 connection(s)
I20260923 14:38:35.862268 16252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:38:35.864558 16320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:35.867790 16320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:38:35.869462 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 16252
I20260923 14:38:35.869586 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:38:35.869839 2920 external_mini_cluster.cc:1428] Setting key 580f8edf89bc6d9f3cf107a9fb0c6817
Sep 23 14:38:35 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.61@KRBTEST.COM" created.
I20260923 14:38:35.881289 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:35.869748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:34051 (local address 127.0.0.1:60714)
0923 14:38:35.870314 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:35.870357 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:35.871082 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:35.871666 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:35.871671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:35.872040 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:35.872235 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:35.872246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:35.873292 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:35.873299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:35.874536 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:35.874550 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:35.875129 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:35.875133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:35.875261 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:35.875790 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:35.875808 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:35.877212 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:35.878918 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:35.878927 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:35.878935 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:35.879133 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:35.879432 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:35.879435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:35.879436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:35.879537 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:35.880118 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:35.880122 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:35.880224 (+ 102us) client_negotiation.cc:769] Sending connection context
0923 14:38:35.880344 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 14:38:35.880544 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"spinlock_wait_cycles":39680,"thread_start_us":143,"threads_started":1}
I20260923 14:38:35.881289 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:35.869748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.2.218.62:34051)
0923 14:38:35.870182 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:35.870199 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:35.870482 (+ 283us) server_negotiation.cc:407] Connection header received
0923 14:38:35.871352 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:35.871356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:35.871502 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:35.871667 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:35.872390 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:35.873140 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:35.874663 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:35.874983 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:35.875086 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:35.877345 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:35.877349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:35.877357 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:35.877373 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:35.878785 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:35.879232 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:35.879238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:35.879239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:35.879338 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:35.879617 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:35.879622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:35.879624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:35.879991 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:35.880121 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:35.880442 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 14:38:35.880636 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":257,"thread_start_us":125,"threads_started":1}
I20260923 14:38:35.883463 16320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:38:35.884153 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:35.883852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34915 (local address 127.0.0.1:58252)
0923 14:38:35.883986 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:35.884078 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 14:38:35.884334 16271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34915: connect: Connection refused (error 111)
I20260923 14:38:35.885094 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:35.884861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34915 (local address 127.0.0.1:58258)
0923 14:38:35.884969 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:35.885032 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 14:38:35.885221 16320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.61:34915: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34915: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab.
Entry for principal kudu/127.2.218.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab.
I20260923 14:38:35.892843 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.61:34915
--webserver_interface=127.2.218.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab
--principal=kudu/127.2.218.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:34051,127.2.218.61:34915,127.2.218.60:39351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:38:35.934456 16320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } attempt: 1
I20260923 14:38:35.935179 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:35.934921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34915 (local address 127.0.0.1:58262)
0923 14:38:35.935057 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:35.935114 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 14:38:35.935806 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:35.935611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34915 (local address 127.0.0.1:58266)
0923 14:38:35.935707 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:35.935749 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 14:38:35.935920 16320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.61:34915: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34915: connect: Connection refused (error 111)
W20260923 14:38:35.988125 16327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:35.988309 16327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:35.988341 16327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:35.991617 16327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:38:35.991653 16327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:35.991669 16327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:35.991683 16327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:38:35.991695 16327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:38:35.995769 16327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:34051,127.2.218.61:34915,127.2.218.60:39351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab
--principal=kudu/127.2.218.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.61:34915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.218.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:35.996480 16327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:35.997316 16327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:38:36.004093 16320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } attempt: 2
W20260923 14:38:36.004729 16335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:36.004734 16332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:36.004885 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.004575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34915 (local address 127.0.0.1:58272)
0923 14:38:36.004712 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.004777 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 14:38:36.004860 16333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:36.005195 16327 server_base.cc:1034] running on GCE node
I20260923 14:38:36.005604 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.005379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34915 (local address 127.0.0.1:58278)
0923 14:38:36.005489 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.005538 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 14:38:36.005733 16320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.61:34915: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34915: connect: Connection refused (error 111)
I20260923 14:38:36.005787 16327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:36.006755 16327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:36.007899 16327 hybrid_clock.cc:648] HybridClock initialized: now 1790174316007888 us; error 24 us; skew 500 ppm
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:36.010368 16327 init.cc:377] Logged in from keytab as kudu/127.2.218.61@KRBTEST.COM (short username kudu)
I20260923 14:38:36.011130 16327 webserver.cc:460] Webserver started at http://127.2.218.61:34447/ using document root <none> and password file <none>
I20260923 14:38:36.011590 16327 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:36.011643 16327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:36.011865 16327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:36.013550 16327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/data/instance:
uuid: "a69665c325234239aaf1de3906cebb7d"
format_stamp: "Formatted at 2026-09-23 14:38:36 on dist-test-slave-zl47"
server_key: "0d024d2587cbb3ea7444061a2060f3e1"
server_key_iv: "af4f411b9f940f2f236d66178000c1c5"
server_key_version: "encryptionkey@0"
I20260923 14:38:36.014137 16327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/wal/instance:
uuid: "a69665c325234239aaf1de3906cebb7d"
format_stamp: "Formatted at 2026-09-23 14:38:36 on dist-test-slave-zl47"
server_key: "0d024d2587cbb3ea7444061a2060f3e1"
server_key_iv: "af4f411b9f940f2f236d66178000c1c5"
server_key_version: "encryptionkey@0"
I20260923 14:38:36.017170 16327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:38:36.019304 16342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:36.020394 16327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:38:36.020529 16327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/wal
uuid: "a69665c325234239aaf1de3906cebb7d"
format_stamp: "Formatted at 2026-09-23 14:38:36 on dist-test-slave-zl47"
server_key: "0d024d2587cbb3ea7444061a2060f3e1"
server_key_iv: "af4f411b9f940f2f236d66178000c1c5"
server_key_version: "encryptionkey@0"
I20260923 14:38:36.020637 16327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:36.046412 16327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:36.048619 16327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:36.048787 16327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:36.054864 16327 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.61:34915
I20260923 14:38:36.054896 16394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.61:34915 every 8 connection(s)
I20260923 14:38:36.055657 16327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
I20260923 14:38:36.057629 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 16327
I20260923 14:38:36.057751 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-1/wal/instance
I20260923 14:38:36.057993 2920 external_mini_cluster.cc:1428] Setting key 2728670fade199c05e6e2c300a4ad9cb
I20260923 14:38:36.058032 16395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:36.061703 16395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.218.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.60@KRBTEST.COM" created.
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:36.074962 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.063305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.2.218.62:34051)
0923 14:38:36.063433 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.063436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.064270 (+ 834us) server_negotiation.cc:407] Connection header received
0923 14:38:36.065034 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.065038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.065096 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.065188 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.066330 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.066833 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.067814 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.068072 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.068175 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.070532 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.070539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.070545 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.070576 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.072887 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.073312 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.073319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.073322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.073416 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.073699 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.073703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.073704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.073957 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.074084 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.074578 (+ 494us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.074726 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:38:36.075428 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.063720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:34051 (local address 127.0.0.1:60718)
0923 14:38:36.064141 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.064176 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:36.064857 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:36.065330 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:36.065335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:36.065696 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:36.066184 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.066198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.066958 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.066966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.067691 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.067699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.068203 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.068206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.068380 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.068922 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:36.068935 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:36.070379 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:36.072998 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.073002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.073010 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.073218 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.073507 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.073509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.073510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.073613 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.074073 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:36.074077 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:36.074348 (+ 271us) client_negotiation.cc:769] Sending connection context
0923 14:38:36.074478 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 14:38:36.074686 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
I20260923 14:38:36.076582 16395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab.
Entry for principal kudu/127.2.218.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab.
I20260923 14:38:36.083083 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.60:39351
--webserver_interface=127.2.218.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab
--principal=kudu/127.2.218.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:34975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:34051,127.2.218.61:34915,127.2.218.60:39351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:38:36.085314 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.077041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34915 (local address 127.0.0.1:58284)
0923 14:38:36.077116 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.077150 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:36.077248 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:36.077607 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:36.077610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:36.077720 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:36.077863 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.077870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.078867 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.078873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.079437 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.079446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.080003 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.080006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.080092 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.080466 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:36.080476 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:36.081804 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:36.083773 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.083787 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.083792 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.084056 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.084416 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.084419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.084420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.084497 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.084964 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:36.084966 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:36.085038 (+ 72us) client_negotiation.cc:769] Sending connection context
0923 14:38:36.085123 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 14:38:36.085210 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 14:38:36.085647 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.077091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.2.218.61:34915)
0923 14:38:36.077331 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.077340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.077355 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:38:36.077383 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.077386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.077497 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.077583 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.077956 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.078731 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.079549 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.079910 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.079990 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.081930 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.081946 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.081955 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.081974 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.083661 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.084186 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.084193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.084196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.084288 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.084586 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.084591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.084593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.084864 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.084987 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.085241 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.085385 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260923 14:38:36.087494 16395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:38:36.088127 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.087898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57332)
0923 14:38:36.088000 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.088077 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 14:38:36.088384 16345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
I20260923 14:38:36.089169 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.088977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57342)
0923 14:38:36.089078 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.089122 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 14:38:36.089390 16395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:39351: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
I20260923 14:38:36.096876 16320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } attempt: 3
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
I20260923 14:38:36.104733 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.097223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34915 (local address 127.0.0.1:58300)
0923 14:38:36.097346 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.097363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:36.097450 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:36.097682 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:36.097685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:36.097853 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:36.097946 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.097951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.098692 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.098697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.099174 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.099182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.100023 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.100027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.100132 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.100575 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:36.100586 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:36.101669 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:36.103335 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.103338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.103340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.103525 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.103808 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.103811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.103813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.103896 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.104245 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:36.104248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:36.104343 (+ 95us) client_negotiation.cc:769] Sending connection context
0923 14:38:36.104446 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 14:38:36.104569 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 14:38:36.104740 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.097290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.2.218.61:34915)
0923 14:38:36.097423 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.097427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.097440 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:38:36.097540 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.097543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.097587 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.097668 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.098057 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.098582 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.099308 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.099906 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.100021 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.101800 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.101803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.101806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.101837 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.103185 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.103633 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.103636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.103638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.103713 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.103988 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.103993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.103994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.104148 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.104252 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.104505 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.104603 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:38:36.105413 16320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:38:36.106025 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.105818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57346)
0923 14:38:36.105921 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.105975 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 14:38:36.106792 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.106558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57354)
0923 14:38:36.106701 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.106743 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260923 14:38:36.106967 16320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:39351: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
I20260923 14:38:36.138129 16320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } attempt: 1
I20260923 14:38:36.138624 16395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } attempt: 1
I20260923 14:38:36.138723 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.138486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57368)
0923 14:38:36.138605 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.138659 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 14:38:36.139171 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.138939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57372)
0923 14:38:36.139027 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.139105 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 14:38:36.139438 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.139192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57386)
0923 14:38:36.139343 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.139389 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260923 14:38:36.139766 16320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:39351: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
I20260923 14:38:36.139815 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.139615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57402)
0923 14:38:36.139709 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.139752 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 14:38:36.139930 16395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:39351: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
W20260923 14:38:36.184907 16401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:36.185112 16401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:36.185144 16401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:36.188475 16401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:38:36.188516 16401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:36.188534 16401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:36.188570 16401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:38:36.188599 16401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:38:36.192848 16401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:34975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:34051,127.2.218.61:34915,127.2.218.60:39351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab
--principal=kudu/127.2.218.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.60:39351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.218.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:36.193602 16401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:36.194439 16401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:36.201326 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:36.201377 16408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:36.201458 16401 server_base.cc:1034] running on GCE node
W20260923 14:38:36.201709 16407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:36.202270 16401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:36.203150 16401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:36.204294 16401 hybrid_clock.cc:648] HybridClock initialized: now 1790174316204276 us; error 32 us; skew 500 ppm
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:36.206626 16401 init.cc:377] Logged in from keytab as kudu/127.2.218.60@KRBTEST.COM (short username kudu)
I20260923 14:38:36.207362 16401 webserver.cc:460] Webserver started at http://127.2.218.60:33147/ using document root <none> and password file <none>
I20260923 14:38:36.207988 16401 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:36.208065 16401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:36.208078 16395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } attempt: 2
I20260923 14:38:36.208254 16401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:36.208812 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.208494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57416)
0923 14:38:36.208644 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.208709 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 14:38:36.209437 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.209264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57430)
0923 14:38:36.209358 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.209395 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 14:38:36.209677 16395 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:39351: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
I20260923 14:38:36.210378 16401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/data/instance:
uuid: "58e5dacd43b649adb43fdcb71cd761cd"
format_stamp: "Formatted at 2026-09-23 14:38:36 on dist-test-slave-zl47"
server_key: "9130c89cdf86932c064711f02bb3d197"
server_key_iv: "1ec782f5ba8669e55bc947045427ffdd"
server_key_version: "encryptionkey@0"
I20260923 14:38:36.210887 16401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/wal/instance:
uuid: "58e5dacd43b649adb43fdcb71cd761cd"
format_stamp: "Formatted at 2026-09-23 14:38:36 on dist-test-slave-zl47"
server_key: "9130c89cdf86932c064711f02bb3d197"
server_key_iv: "1ec782f5ba8669e55bc947045427ffdd"
server_key_version: "encryptionkey@0"
I20260923 14:38:36.213970 16401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:38:36.214934 16320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } attempt: 2
I20260923 14:38:36.215591 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.215313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57442)
0923 14:38:36.215447 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.215509 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 14:38:36.216243 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.216027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57450)
0923 14:38:36.216133 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.216186 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 14:38:36.216394 16320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:39351: Network error: Client connection negotiation failed: client connection to 127.2.218.60:39351: connect: Connection refused (error 111)
I20260923 14:38:36.216455 16417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:36.217406 16401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:36.217525 16401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/wal
uuid: "58e5dacd43b649adb43fdcb71cd761cd"
format_stamp: "Formatted at 2026-09-23 14:38:36 on dist-test-slave-zl47"
server_key: "9130c89cdf86932c064711f02bb3d197"
server_key_iv: "1ec782f5ba8669e55bc947045427ffdd"
server_key_version: "encryptionkey@0"
I20260923 14:38:36.217619 16401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:36.231070 16401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:36.233422 16401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:36.233604 16401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:36.240155 16401 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.60:39351
I20260923 14:38:36.240207 16469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.60:39351 every 8 connection(s)
I20260923 14:38:36.241007 16401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
I20260923 14:38:36.243427 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:36.246388 16470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:38:36.248674 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 16401
I20260923 14:38:36.248777 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790174173335390-2920-0/minicluster-data/master-2/wal/instance
I20260923 14:38:36.249037 2920 external_mini_cluster.cc:1428] Setting key bb1ae2b6f5acb9062c6d3bda0199fbbd
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
I20260923 14:38:36.258272 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.247779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.2.218.62:34051)
0923 14:38:36.247890 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.247893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.248541 (+ 648us) server_negotiation.cc:407] Connection header received
0923 14:38:36.249547 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.249550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.249601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.249684 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.250775 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.251400 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.252672 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.252962 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.253050 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.255084 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.255087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.255089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.255117 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.256392 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.256933 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.256936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.256937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.257003 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.257311 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.257314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.257315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.257451 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.257516 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.257979 (+ 463us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.258100 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:38:36.258971 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.248060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:34051 (local address 127.0.0.1:60722)
0923 14:38:36.248414 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.248450 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:36.249339 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:36.249866 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:36.249876 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:36.250258 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:36.250640 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.250652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.251503 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.251506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.252547 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.252554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.253077 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.253079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.253195 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.253634 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:36.253651 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:36.254958 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:36.256492 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.256502 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.256511 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.256844 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.257088 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.257092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.257094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.257227 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.257534 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:36.257539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:36.257756 (+ 217us) client_negotiation.cc:769] Sending connection context
0923 14:38:36.257917 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 14:38:36.258201 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
I20260923 14:38:36.259042 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.249634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.2.218.60:39351)
0923 14:38:36.249884 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.249893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.250068 (+ 175us) server_negotiation.cc:407] Connection header received
0923 14:38:36.250202 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.250209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.250368 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.250503 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.251254 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.252391 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.253351 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.253757 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.253853 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.255948 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.255954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.255964 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.255990 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.257533 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.257868 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.257875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.257878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.257950 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.258206 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.258209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.258210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.258426 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.258585 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.258773 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.258881 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"spinlock_wait_cycles":4480,"thread_start_us":75,"threads_started":1}
I20260923 14:38:36.260070 16470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:36.267584 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.260456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34915 (local address 127.0.0.1:58310)
0923 14:38:36.260573 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.260585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:36.260642 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:36.260938 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:36.260942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:36.261115 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:36.261218 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.261226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.262001 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.262003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.262636 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.262645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.263046 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.263048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.263108 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.263473 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:36.263486 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:36.264598 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:36.266356 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.266358 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.266359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.266515 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.266779 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.266781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.266782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.266844 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.267168 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:36.267172 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:36.267254 (+ 82us) client_negotiation.cc:769] Sending connection context
0923 14:38:36.267354 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 14:38:36.267462 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 14:38:36.267645 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.260529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58310 (local address 127.2.218.61:34915)
0923 14:38:36.260632 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.260636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.260648 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:38:36.260721 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.260723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.260759 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.260827 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.261352 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.261908 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.262743 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.262924 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.263032 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.264715 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.264717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.264719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.264755 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.266252 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.266616 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.266620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.266622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.266697 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.266924 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.266927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.266930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.267084 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.267145 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.267419 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.267517 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:38:36.268271 16470 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:38:36.268878 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.260775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.2.218.62:34051)
0923 14:38:36.260859 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.260863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.260876 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:38:36.260980 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.260982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.261010 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.261072 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.261613 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.262207 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.262982 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.263231 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.263323 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.265538 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.265542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.265546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.265568 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.266848 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.267210 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.267212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.267213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.267267 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.267517 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.267520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.267521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.267939 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.268132 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.268336 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.268453 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
I20260923 14:38:36.275314 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.268647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57478)
0923 14:38:36.268745 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.268758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:36.268834 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:36.269027 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:36.269029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:36.269163 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:36.269231 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.269236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.270124 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.270127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.270620 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.270629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.271021 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.271025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.271093 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.271480 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:36.271490 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:36.272521 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:36.273865 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.273868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.273870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.274058 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.274342 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.274345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.274346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.274420 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.274792 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:36.274795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:36.274883 (+ 88us) client_negotiation.cc:769] Sending connection context
0923 14:38:36.275019 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 14:38:36.275151 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 14:38:36.275425 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.268722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.2.218.60:39351)
0923 14:38:36.268828 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.268832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.268844 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:38:36.268910 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.268914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.268952 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.269019 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.269364 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.270013 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.270725 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.270930 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.270997 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.272625 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.272627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.272629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.272647 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.273734 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.274187 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.274190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.274192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.274258 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.274511 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.274514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.274515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.274699 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.274783 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.275067 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.275219 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":43}
I20260923 14:38:36.277375 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.269816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.2.218.61:34915)
0923 14:38:36.269908 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.269911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.269929 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:38:36.269984 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.269986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.270011 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.270108 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.270585 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.271108 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.271920 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.272093 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.272173 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.274411 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.274414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.274416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.274437 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.275980 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.276385 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.276387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.276388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.276484 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.276732 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.276735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.276736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.276928 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.277114 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.277174 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.277262 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 14:38:36.278472 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd: Bootstrap starting.
I20260923 14:38:36.280012 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:38:36.280645 16470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:36.281471 16470 log.cc:824] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:36.283504 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd: No bootstrap required, opened a new log
I20260923 14:38:36.287377 16470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } }
I20260923 14:38:36.287724 16470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:36.287797 16470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e5dacd43b649adb43fdcb71cd761cd, State: Initialized, Role: FOLLOWER
I20260923 14:38:36.288313 16470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } }
I20260923 14:38:36.289683 16470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:38:36.290673 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.282373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.2.218.60:39351)
0923 14:38:36.282455 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.282459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.282672 (+ 213us) server_negotiation.cc:407] Connection header received
0923 14:38:36.282767 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.282768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.282797 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.282882 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.283652 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.284314 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.285309 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.285600 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.285658 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.287104 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.287110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.287115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.287137 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.288707 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.289172 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.289177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.289179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.289259 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.289807 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.289810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.289812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.290020 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.290130 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.290444 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.290550 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 14:38:36.291626 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.282063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.2.218.61:34915)
0923 14:38:36.282474 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.282477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.283071 (+ 594us) server_negotiation.cc:407] Connection header received
0923 14:38:36.283167 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.283171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.283212 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.283286 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.284073 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.284692 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.286374 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.286726 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.287149 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.288254 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.288257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.288259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.288282 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.289690 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.290143 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.290148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.290152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.290227 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.290527 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.290530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.290532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.290695 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.291366 (+ 671us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.291429 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.291518 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354}
I20260923 14:38:36.292316 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } } }
I20260923 14:38:36.292778 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:38:36.292923 16484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:38:36.293066 16484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } }
I20260923 14:38:36.293342 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.282036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.2.218.62:34051)
0923 14:38:36.282153 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.282159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.283372 (+ 1213us) server_negotiation.cc:407] Connection header received
0923 14:38:36.283527 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.283530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.283573 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.283617 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.284487 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.285131 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.286602 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.286903 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.286972 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.289472 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.289475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.289477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.289499 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.291104 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.291613 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.291616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.291617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.291697 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.292514 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.292517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.292519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.292684 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.292768 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.293092 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.293187 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:38:36.294003 16484 leader_election.cc:288] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d85ed1fa7c8848e3a49d6c3d5dc48cef (127.2.218.62:34051), a69665c325234239aaf1de3906cebb7d (127.2.218.61:34915)
W20260923 14:38:36.294737 16495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:38:36.294809 16495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:38:36.293500 16401 master_runner.cc:186] Error getting master registration for 127.2.218.62:34051: OK, instance_id { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" instance_seqno: 1790174315859690 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:38:36.296190 16401 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:38:36.300791 16395 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } attempt: 3
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174316, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
I20260923 14:38:36.308436 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.301143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57486)
0923 14:38:36.301286 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.301304 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:36.301367 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:36.301543 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:36.301545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:36.301683 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:36.301743 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.301747 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.302564 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.302567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.303139 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.303148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.303734 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.303737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.303824 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.304204 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:36.304215 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:36.305556 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:36.307204 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.307209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.307211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.307413 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.307651 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.307655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.307657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.307732 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.308038 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:36.308041 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:36.308117 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 14:38:36.308203 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 14:38:36.308311 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 14:38:36.308465 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.301217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.2.218.60:39351)
0923 14:38:36.301312 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.301317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.301334 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:38:36.301429 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.301432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.301462 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.301521 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.301842 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.302472 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.303230 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.303477 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.303561 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.305674 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.305677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.305679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.305705 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.307092 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.307496 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.307499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.307500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.307569 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.307805 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.307808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.307809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.307951 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.308043 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.308270 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.308361 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:38:36.311188 16395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d: Bootstrap starting.
I20260923 14:38:36.313459 16395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:36.314208 16395 log.cc:824] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:36.315517 16320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } attempt: 3
I20260923 14:38:36.316363 16395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d: No bootstrap required, opened a new log
I20260923 14:38:36.319947 16395 raft_consensus.cc:348] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } }
Sep 23 14:38:36 dist-test-slave-zl47 krb5kdc[16239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174315, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
I20260923 14:38:36.320986 16395 raft_consensus.cc:374] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:36.321063 16395 raft_consensus.cc:729] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69665c325234239aaf1de3906cebb7d, State: Initialized, Role: FOLLOWER
I20260923 14:38:36.321477 16395 consensus_queue.cc:260] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } }
I20260923 14:38:36.322777 16395 sys_catalog.cc:564] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:38:36.322832 16369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58e5dacd43b649adb43fdcb71cd761cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69665c325234239aaf1de3906cebb7d" is_pre_election: true
I20260923 14:38:36.323043 16369 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58e5dacd43b649adb43fdcb71cd761cd in term 0.
I20260923 14:38:36.323460 16421 leader_election.cc:302] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e5dacd43b649adb43fdcb71cd761cd, a69665c325234239aaf1de3906cebb7d; no voters:
I20260923 14:38:36.322706 16498 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } } }
I20260923 14:38:36.323858 16498 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:38:36.324103 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.315839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:39351 (local address 127.0.0.1:57492)
0923 14:38:36.315971 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:36.315983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:36.316044 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:36.316277 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:36.316279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:36.316431 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:36.316511 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.316517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.317351 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.317354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.318050 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:36.318059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.318620 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.318623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:36.318710 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.319164 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:36.319176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:36.320391 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:36.322596 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.322599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.322601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.322824 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.323122 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:36.323125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:36.323127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:36.323200 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:36.323666 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:36.323669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:36.323761 (+ 92us) client_negotiation.cc:769] Sending connection context
0923 14:38:36.323860 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 14:38:36.323968 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 14:38:36.324777 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:36.315896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.2.218.60:39351)
0923 14:38:36.316010 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:36.316013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:36.316025 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:38:36.316109 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:36.316133 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:36.316167 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:36.316275 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:36.316609 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.317234 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.318208 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:36.318480 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:36.318590 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:36.321012 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:36.321016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:36.321018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:36.321053 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:36.322503 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.322953 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.322955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.322957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.323031 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:36.323370 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:36.323373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:36.323375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:36.323550 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:36.323627 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:36.324568 (+ 941us) server_negotiation.cc:299] Negotiation successful
0923 14:38:36.324661 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:38:36.325244 16483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:38:36.325398 16483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:38:36.325518 16483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:36.327160 16483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } }
W20260923 14:38:36.327344 16509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:38:36.327407 16509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:38:36.328131 16483 leader_election.cc:288] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [CANDIDATE]: Term 1 election: Requested vote from peers d85ed1fa7c8848e3a49d6c3d5dc48cef (127.2.218.62:34051), a69665c325234239aaf1de3906cebb7d (127.2.218.61:34915)
I20260923 14:38:36.328388 16369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58e5dacd43b649adb43fdcb71cd761cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69665c325234239aaf1de3906cebb7d"
I20260923 14:38:36.328495 16369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:36.325729 16327 master_runner.cc:186] Error getting master registration for 127.2.218.62:34051: OK, instance_id { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" instance_seqno: 1790174315859690 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:38:36.329085 16327 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:38:36.329476 16369 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58e5dacd43b649adb43fdcb71cd761cd in term 1.
I20260923 14:38:36.329710 16421 leader_election.cc:302] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e5dacd43b649adb43fdcb71cd761cd, a69665c325234239aaf1de3906cebb7d; no voters:
I20260923 14:38:36.329928 16483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:38:36.330140 16483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [term 1 LEADER]: Becoming Leader. State: Replica: 58e5dacd43b649adb43fdcb71cd761cd, State: Running, Role: LEADER
I20260923 14:38:36.330634 16483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } }
I20260923 14:38:36.332294 16320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef: Bootstrap starting.
I20260923 14:38:36.334511 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e5dacd43b649adb43fdcb71cd761cd. Latest consensus state: current_term: 1 leader_uuid: "58e5dacd43b649adb43fdcb71cd761cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } } }
I20260923 14:38:36.334628 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:36.335238 16320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:36.335951 16320 log.cc:824] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:36.346292 16512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:38:36.347070 16512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:38:36.350519 16320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef: No bootstrap required, opened a new log
I20260923 14:38:36.352830 16369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [term 1 FOLLOWER]: Refusing update from remote peer 58e5dacd43b649adb43fdcb71cd761cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:38:36.353441 16484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:38:36.354760 16320 raft_consensus.cc:348] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } }
I20260923 14:38:36.355011 16320 raft_consensus.cc:374] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:36.355053 16320 raft_consensus.cc:729] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85ed1fa7c8848e3a49d6c3d5dc48cef, State: Initialized, Role: FOLLOWER
I20260923 14:38:36.355562 16320 consensus_queue.cc:260] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } }
I20260923 14:38:36.357103 16498 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e5dacd43b649adb43fdcb71cd761cd. Latest consensus state: current_term: 1 leader_uuid: "58e5dacd43b649adb43fdcb71cd761cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } } }
I20260923 14:38:36.357198 16498 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:38:36.357270 16294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58e5dacd43b649adb43fdcb71cd761cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" is_pre_election: true
I20260923 14:38:36.357486 16294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58e5dacd43b649adb43fdcb71cd761cd in term 0.
I20260923 14:38:36.357951 16293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "58e5dacd43b649adb43fdcb71cd761cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef"
I20260923 14:38:36.358054 16293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:36.359143 16293 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58e5dacd43b649adb43fdcb71cd761cd in term 1.
I20260923 14:38:36.359710 16292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [term 1 FOLLOWER]: Refusing update from remote peer 58e5dacd43b649adb43fdcb71cd761cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:38:36.357931 16320 sys_catalog.cc:564] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:38:36.360172 16483 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:38:36.363708 16515 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } } }
I20260923 14:38:36.363842 16515 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:38:36.363950 16515 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e5dacd43b649adb43fdcb71cd761cd. Latest consensus state: current_term: 1 leader_uuid: "58e5dacd43b649adb43fdcb71cd761cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } } }
I20260923 14:38:36.364025 16515 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:38:36.364876 16483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58e5dacd43b649adb43fdcb71cd761cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } } }
I20260923 14:38:36.364970 16483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:36.365501 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58e5dacd43b649adb43fdcb71cd761cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } } }
I20260923 14:38:36.365590 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:36.366197 16498 sys_catalog.cc:455] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58e5dacd43b649adb43fdcb71cd761cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } } }
I20260923 14:38:36.366282 16498 sys_catalog.cc:458] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:38:36.366551 16529 mvcc.cc:205] Tried to move back new op lower bound from 7332553999774855168 to 7332553999692578816. Current Snapshot: MvccSnapshot[applied={T|T < 7332553999774855168}]
I20260923 14:38:36.366703 16523 sys_catalog.cc:455] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58e5dacd43b649adb43fdcb71cd761cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85ed1fa7c8848e3a49d6c3d5dc48cef" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34051 } } peers { permanent_uuid: "a69665c325234239aaf1de3906cebb7d" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34915 } } peers { permanent_uuid: "58e5dacd43b649adb43fdcb71cd761cd" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 39351 } } }
I20260923 14:38:36.366793 16523 sys_catalog.cc:458] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:38:36.369897 16516 mvcc.cc:205] Tried to move back new op lower bound from 7332553999774855168 to 7332553999692578816. Current Snapshot: MvccSnapshot[applied={T|T < 7332553999774855168}]
I20260923 14:38:36.371027 16512 catalog_manager.cc:1347] Generated new cluster ID: 564be36616a74d8ebcc2b109ae12bf93
I20260923 14:38:36.371084 16512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260923 14:38:36.372563 16531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:38:36.372635 16531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:38:36.399390 16512 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:38:36.400410 16512 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:38:36.414777 16512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58e5dacd43b649adb43fdcb71cd761cd: Generated new TSK 0
I20260923 14:38:36.415412 16512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:38:37.328579 16509 catalog_manager.cc:1259] Loaded cluster ID: 564be36616a74d8ebcc2b109ae12bf93
I20260923 14:38:37.328678 16509 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d: loading cluster ID for follower catalog manager: success
I20260923 14:38:37.330170 16509 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d: acquiring CA information for follower catalog manager: success
I20260923 14:38:37.331184 16509 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a69665c325234239aaf1de3906cebb7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:38:37.373776 16531 catalog_manager.cc:1259] Loaded cluster ID: 564be36616a74d8ebcc2b109ae12bf93
I20260923 14:38:37.373836 16531 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef: loading cluster ID for follower catalog manager: success
I20260923 14:38:37.375356 16531 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef: acquiring CA information for follower catalog manager: success
I20260923 14:38:37.376367 16531 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d85ed1fa7c8848e3a49d6c3d5dc48cef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:38:37.471271 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 16252
I20260923 14:38:37.477166 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 16327
I20260923 14:38:37.482560 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 16401
2026-09-23T14:38:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2804 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:38:37 dist-test-slave-zl47 krb5kdc[16543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:38:37 dist-test-slave-zl47 krb5kdc[16543](info): set up 2 sockets
Sep 23 14:38:37 dist-test-slave-zl47 krb5kdc[16543](info): commencing operation
krb5kdc: starting...
W20260923 14:38:38.398307 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.896s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:38:38 dist-test-slave-zl47 krb5kdc[16543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:38:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:38:38.501158 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:39255
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:39255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:38.595387 16556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:38.595641 16556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:38.595707 16556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:38.599076 16556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:38:38.599157 16556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:38.599191 16556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:38.599242 16556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:38:38.599274 16556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:38:38.603927 16556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:39255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:39255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:38.604874 16556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:38.605736 16556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:38.613303 16562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:38.613332 16564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:38.613332 16561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:38.613546 16556 server_base.cc:1034] running on GCE node
I20260923 14:38:38.614064 16556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:38.615041 16556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:38.616202 16556 hybrid_clock.cc:648] HybridClock initialized: now 1790174318616182 us; error 37 us; skew 500 ppm
Sep 23 14:38:38 dist-test-slave-zl47 krb5kdc[16543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174318, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:38.618611 16556 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:38:38.619405 16556 webserver.cc:460] Webserver started at http://127.2.218.62:32951/ using document root <none> and password file <none>
I20260923 14:38:38.619896 16556 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:38.619951 16556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:38.620146 16556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:38.621984 16556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "124e74fd69ff425081d6c5bee839502e"
format_stamp: "Formatted at 2026-09-23 14:38:38 on dist-test-slave-zl47"
server_key: "7516f8daf9875059fa0dd2c08f7a90b3"
server_key_iv: "62ec98090d08c79550f46ef2bd1d0f5a"
server_key_version: "encryptionkey@0"
I20260923 14:38:38.622558 16556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "124e74fd69ff425081d6c5bee839502e"
format_stamp: "Formatted at 2026-09-23 14:38:38 on dist-test-slave-zl47"
server_key: "7516f8daf9875059fa0dd2c08f7a90b3"
server_key_iv: "62ec98090d08c79550f46ef2bd1d0f5a"
server_key_version: "encryptionkey@0"
I20260923 14:38:38.625710 16556 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:38:38.627908 16571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:38.628978 16556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:38.629113 16556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "124e74fd69ff425081d6c5bee839502e"
format_stamp: "Formatted at 2026-09-23 14:38:38 on dist-test-slave-zl47"
server_key: "7516f8daf9875059fa0dd2c08f7a90b3"
server_key_iv: "62ec98090d08c79550f46ef2bd1d0f5a"
server_key_version: "encryptionkey@0"
I20260923 14:38:38.629216 16556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:38.644598 16556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:38.647046 16556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:38.647233 16556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:38.653941 16556 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:39255
I20260923 14:38:38.653963 16623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:39255 every 8 connection(s)
I20260923 14:38:38.654686 16556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:38:38.655830 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 16556
I20260923 14:38:38.655910 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:38:38.656167 2920 external_mini_cluster.cc:1428] Setting key 5f3cd2f0d3ad7a73d027f8eaa550ba99
I20260923 14:38:38.657173 16624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:38.662627 16624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e: Bootstrap starting.
Sep 23 14:38:38 dist-test-slave-zl47 krb5kdc[16543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:38.664670 16624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:38.665478 16624 log.cc:824] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:38.667263 16624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e: No bootstrap required, opened a new log
I20260923 14:38:38.668653 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:38.657326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33788 (local address 127.2.218.62:39255)
0923 14:38:38.657674 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:38.657690 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:38.657724 (+ 34us) server_negotiation.cc:407] Connection header received
0923 14:38:38.658303 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:38.658317 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:38.658596 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:38.658836 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:38.659640 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:38.660518 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:38.661617 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:38.662021 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:38.662183 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:38.664396 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:38.664400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:38.664410 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:38.664431 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:38.665990 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:38.666404 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:38.666408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:38.666413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:38.666523 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:38.666774 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:38.666777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:38.666781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:38.667186 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:38.667369 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:38.667658 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 14:38:38.667907 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":135,"threads_started":1}
I20260923 14:38:38.671188 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:38:38.671105 16624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e74fd69ff425081d6c5bee839502e" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39255 } }
I20260923 14:38:38.671342 16624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:38.671425 16624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124e74fd69ff425081d6c5bee839502e, State: Initialized, Role: FOLLOWER
I20260923 14:38:38.671835 16624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e74fd69ff425081d6c5bee839502e" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39255 } }
I20260923 14:38:38.671947 16624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:38:38.672024 16624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:38:38.672120 16624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:38.673022 16624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e74fd69ff425081d6c5bee839502e" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39255 } }
I20260923 14:38:38.673261 16624 leader_election.cc:302] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 124e74fd69ff425081d6c5bee839502e; no voters:
I20260923 14:38:38.673473 16624 leader_election.cc:288] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:38:38.673615 16629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:38:38.673866 16629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [term 1 LEADER]: Becoming Leader. State: Replica: 124e74fd69ff425081d6c5bee839502e, State: Running, Role: LEADER
I20260923 14:38:38.674190 16629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e74fd69ff425081d6c5bee839502e" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39255 } }
I20260923 14:38:38.674490 16624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:38:38.675697 16631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 124e74fd69ff425081d6c5bee839502e. Latest consensus state: current_term: 1 leader_uuid: "124e74fd69ff425081d6c5bee839502e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e74fd69ff425081d6c5bee839502e" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39255 } } }
I20260923 14:38:38.675836 16631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:38.676105 16630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "124e74fd69ff425081d6c5bee839502e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124e74fd69ff425081d6c5bee839502e" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39255 } } }
I20260923 14:38:38.676204 16630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:38.676610 16641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:38:38.678745 16641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:38:38.682898 16641 catalog_manager.cc:1347] Generated new cluster ID: 302d538b73a1401ca282aca4140dc626
I20260923 14:38:38.682957 16641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:38:38.692404 16641 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:38:38.693115 16641 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:38:38.708010 16641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 124e74fd69ff425081d6c5bee839502e: Generated new TSK 0
I20260923 14:38:38.708511 16641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:38:38.726481 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 16556
2026-09-23T14:38:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1242 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:38:38 dist-test-slave-zl47 krb5kdc[16651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:38:38 dist-test-slave-zl47 krb5kdc[16651](info): set up 2 sockets
Sep 23 14:38:38 dist-test-slave-zl47 krb5kdc[16651](info): commencing operation
krb5kdc: starting...
W20260923 14:38:39.638371 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.893s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:38:39 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:38:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:38:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:38:39.738430 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:45857
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:45857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:39.831576 16664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:39.831771 16664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:39.831804 16664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:39.835114 16664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:38:39.835155 16664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:39.835171 16664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:39.835184 16664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:38:39.835197 16664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:38:39.839356 16664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:45857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:45857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:39.840049 16664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:39.840758 16664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:39.847957 16669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:39.847975 16670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:39.847955 16672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:39.848155 16664 server_base.cc:1034] running on GCE node
I20260923 14:38:39.848631 16664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:39.849515 16664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:39.850687 16664 hybrid_clock.cc:648] HybridClock initialized: now 1790174319850675 us; error 26 us; skew 500 ppm
Sep 23 14:38:39 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174319, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:39.852948 16664 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:38:39.853672 16664 webserver.cc:460] Webserver started at http://127.2.218.62:38531/ using document root <none> and password file <none>
I20260923 14:38:39.854171 16664 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:39.854249 16664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:39.854437 16664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:39.856029 16664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "96e737264c3b40e3bcc193b5eeacbb4a"
format_stamp: "Formatted at 2026-09-23 14:38:39 on dist-test-slave-zl47"
server_key: "ccbc395566012442c2019b372b9874db"
server_key_iv: "6d4ba78025a56d3309fcfbc9868ac80f"
server_key_version: "encryptionkey@0"
I20260923 14:38:39.856508 16664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "96e737264c3b40e3bcc193b5eeacbb4a"
format_stamp: "Formatted at 2026-09-23 14:38:39 on dist-test-slave-zl47"
server_key: "ccbc395566012442c2019b372b9874db"
server_key_iv: "6d4ba78025a56d3309fcfbc9868ac80f"
server_key_version: "encryptionkey@0"
I20260923 14:38:39.860494 16664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 14:38:39.862571 16679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:39.863529 16664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:39.863649 16664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "96e737264c3b40e3bcc193b5eeacbb4a"
format_stamp: "Formatted at 2026-09-23 14:38:39 on dist-test-slave-zl47"
server_key: "ccbc395566012442c2019b372b9874db"
server_key_iv: "6d4ba78025a56d3309fcfbc9868ac80f"
server_key_version: "encryptionkey@0"
I20260923 14:38:39.863741 16664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:39.904096 16664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:39.906409 16664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:39.906586 16664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:39.912375 16664 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:45857
I20260923 14:38:39.912599 16731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:45857 every 8 connection(s)
I20260923 14:38:39.913224 16664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:38:39.915481 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:39.919863 16732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Bootstrap starting.
I20260923 14:38:39.921941 16732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:39.922567 16732 log.cc:824] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:39.923254 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 16664
I20260923 14:38:39.923398 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:38:39.923635 2920 external_mini_cluster.cc:1428] Setting key e696137f4c2b0e68e82bb11d01b25ef1
I20260923 14:38:39.924543 16732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: No bootstrap required, opened a new log
I20260923 14:38:39.928630 16732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:39.928853 16732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:39.928921 16732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e737264c3b40e3bcc193b5eeacbb4a, State: Initialized, Role: FOLLOWER
I20260923 14:38:39.929292 16732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:39.929431 16732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:38:39.929488 16732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:38:39.929565 16732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:39.930486 16732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:39.930830 16732 leader_election.cc:302] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96e737264c3b40e3bcc193b5eeacbb4a; no voters:
I20260923 14:38:39.931123 16732 leader_election.cc:288] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:38:39.931209 16737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:38:39.931430 16737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 1 LEADER]: Becoming Leader. State: Replica: 96e737264c3b40e3bcc193b5eeacbb4a, State: Running, Role: LEADER
I20260923 14:38:39.931690 16737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
Sep 23 14:38:39 dist-test-slave-zl47 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:39.932274 16732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:38:39.933531 16739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96e737264c3b40e3bcc193b5eeacbb4a. Latest consensus state: current_term: 1 leader_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } } }
I20260923 14:38:39.933677 16739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:39.933531 16738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } } }
I20260923 14:38:39.933840 16738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:39.934047 16747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:38:39.936781 16747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:38:39.937510 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:39.924559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.2.218.62:45857)
0923 14:38:39.924921 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:39.924936 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:39.924969 (+ 33us) server_negotiation.cc:407] Connection header received
0923 14:38:39.925645 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:39.925664 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:39.925996 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:39.926335 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:39.926901 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:39.928285 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:39.929337 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:39.929970 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:39.930133 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:39.932703 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:39.932709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:39.932725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:39.932751 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:39.934991 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:39.935429 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:39.935433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:39.935437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:39.935524 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:39.935806 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:39.935808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:39.935809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:39.936147 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:39.936285 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:39.936539 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 14:38:39.936792 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":105,"threads_started":1}
I20260923 14:38:39.941327 16747 catalog_manager.cc:1347] Generated new cluster ID: d45d59d8aee54e5e9aaf34d0ed3e002d
I20260923 14:38:39.941381 16747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:38:39.970225 16747 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:38:39.970921 16747 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:38:39.977232 16747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Generated new TSK 0
I20260923 14:38:39.977726 16747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:38:40.007048 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:45857
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:40.125335 16758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:40.125532 16758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:40.125597 16758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:40.129160 16758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:40.129209 16758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:40.129262 16758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:38:40.134162 16758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:40.134965 16758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:40.135818 16758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:40.144192 16763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:40.144166 16766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:40.144166 16764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:40.144483 16758 server_base.cc:1034] running on GCE node
I20260923 14:38:40.144870 16758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:40.145447 16758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:40.146638 16758 hybrid_clock.cc:648] HybridClock initialized: now 1790174320146611 us; error 44 us; skew 500 ppm
Sep 23 14:38:40 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:40.149119 16758 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:38:40.149797 16758 webserver.cc:460] Webserver started at http://127.2.218.1:45935/ using document root <none> and password file <none>
I20260923 14:38:40.150303 16758 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:40.150364 16758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:40.150521 16758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:40.152153 16758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "995089815202416fb2af2712aae7ef4b"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "4789cb2bfded6d3d5d30fdbd5a6fe63a"
server_key_iv: "2a196b8eb4e31c7a65713df5ee403aa4"
server_key_version: "encryptionkey@0"
I20260923 14:38:40.152626 16758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "995089815202416fb2af2712aae7ef4b"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "4789cb2bfded6d3d5d30fdbd5a6fe63a"
server_key_iv: "2a196b8eb4e31c7a65713df5ee403aa4"
server_key_version: "encryptionkey@0"
I20260923 14:38:40.155887 16758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:38:40.158072 16773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:40.159243 16758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:40.159376 16758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "995089815202416fb2af2712aae7ef4b"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "4789cb2bfded6d3d5d30fdbd5a6fe63a"
server_key_iv: "2a196b8eb4e31c7a65713df5ee403aa4"
server_key_version: "encryptionkey@0"
I20260923 14:38:40.159507 16758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:40.177274 16758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:40.179690 16758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:40.179884 16758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:40.180382 16758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:40.181072 16758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:40.181113 16758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:40.181164 16758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:40.181192 16758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:40.188633 16758 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:43471
I20260923 14:38:40.188707 16886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:43471 every 8 connection(s)
I20260923 14:38:40.189436 16758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:38:40.196776 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 16758
I20260923 14:38:40.196920 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:38:40.197181 2920 external_mini_cluster.cc:1428] Setting key 6da3e101d7c74717771ad7977045cc10
Sep 23 14:38:40 dist-test-slave-zl47 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:40.201491 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.191407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:55703 (local address 127.2.218.62:45857)
0923 14:38:40.191540 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:40.191545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:40.192447 (+ 902us) server_negotiation.cc:407] Connection header received
0923 14:38:40.193147 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:40.193149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:40.193193 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:40.193263 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:40.194228 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.194725 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.195703 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.195918 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.196001 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.198218 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:40.198222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:40.198224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:40.198246 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:40.199608 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:40.200160 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:40.200165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:40.200168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:40.200262 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:40.200571 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:40.200576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:40.200578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:40.200806 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:40.200910 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:40.201294 (+ 384us) server_negotiation.cc:299] Negotiation successful
0923 14:38:40.201386 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:38:40.202180 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.191740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:55703)
0923 14:38:40.192328 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:40.192361 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:40.192978 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:40.193403 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:40.193409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:40.193675 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:40.194037 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.194047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.194827 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.194829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.195598 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.195606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.196035 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.196037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.196151 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.196578 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:40.196591 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:40.198041 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:40.199719 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:40.199726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:40.199740 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:40.200051 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:40.200374 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:40.200376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:40.200378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:40.200485 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:40.200906 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:40.200910 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:40.201090 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 14:38:40.201213 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 14:38:40.201417 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":104,"threads_started":1}
I20260923 14:38:40.203222 16887 heartbeater.cc:347] Connected to a master server at 127.2.218.62:45857
I20260923 14:38:40.203439 16887 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:40.203835 16887 heartbeater.cc:511] Master 127.2.218.62:45857 requested a full tablet report, sending...
I20260923 14:38:40.205093 16696 ts_manager.cc:194] Registered new tserver with Master: 995089815202416fb2af2712aae7ef4b (127.2.218.1:43471)
I20260923 14:38:40.206254 16696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:55703
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:38:40.221158 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:45857
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:40.311757 16892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:40.311952 16892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:40.311986 16892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:40.315325 16892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:40.315366 16892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:40.315450 16892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:38:40.319680 16892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:40.320477 16892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:40.321343 16892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:40.329465 16897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:40.329535 16898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:40.329523 16900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:40.329811 16892 server_base.cc:1034] running on GCE node
I20260923 14:38:40.330165 16892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:40.330715 16892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:40.331861 16892 hybrid_clock.cc:648] HybridClock initialized: now 1790174320331839 us; error 35 us; skew 500 ppm
Sep 23 14:38:40 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:40.334630 16892 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:38:40.335449 16892 webserver.cc:460] Webserver started at http://127.2.218.2:44261/ using document root <none> and password file <none>
I20260923 14:38:40.335933 16892 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:40.336014 16892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:40.336185 16892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:40.337911 16892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "9235ee37ce7d4d1788898e1bed052ca8"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "efa63c4afd30dd6ceea517bb143d86e7"
server_key_iv: "3ed740ffd6d3c2b9cc7da4fe2ebed666"
server_key_version: "encryptionkey@0"
I20260923 14:38:40.338521 16892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "9235ee37ce7d4d1788898e1bed052ca8"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "efa63c4afd30dd6ceea517bb143d86e7"
server_key_iv: "3ed740ffd6d3c2b9cc7da4fe2ebed666"
server_key_version: "encryptionkey@0"
I20260923 14:38:40.341794 16892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:38:40.343958 16907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:40.344990 16892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:40.345103 16892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "9235ee37ce7d4d1788898e1bed052ca8"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "efa63c4afd30dd6ceea517bb143d86e7"
server_key_iv: "3ed740ffd6d3c2b9cc7da4fe2ebed666"
server_key_version: "encryptionkey@0"
I20260923 14:38:40.345201 16892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:40.372704 16892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:40.375705 16892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:40.375890 16892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:40.376381 16892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:40.377070 16892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:40.377131 16892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:40.377190 16892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:40.377220 16892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:40.384836 16892 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:40201
I20260923 14:38:40.384862 17020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:40201 every 8 connection(s)
I20260923 14:38:40.385596 16892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
Sep 23 14:38:40 dist-test-slave-zl47 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:40.395856 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 16892
I20260923 14:38:40.395979 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:38:40.396243 2920 external_mini_cluster.cc:1428] Setting key c58c1660d71af746c48f3d913e17accd
I20260923 14:38:40.397441 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.387169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:58741 (local address 127.2.218.62:45857)
0923 14:38:40.387296 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:40.387318 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:40.388013 (+ 695us) server_negotiation.cc:407] Connection header received
0923 14:38:40.389134 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:40.389137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:40.389193 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:40.389265 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:40.390308 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.390812 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.391841 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.392061 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.392166 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.394372 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:40.394375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:40.394377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:40.394399 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:40.395451 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:40.395923 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:40.395927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:40.395928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:40.396037 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:40.396352 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:40.396356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:40.396358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:40.396529 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:40.396628 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:40.397147 (+ 519us) server_negotiation.cc:299] Negotiation successful
0923 14:38:40.397280 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 14:38:40.398171 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.387486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:58741)
0923 14:38:40.387878 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:40.387915 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:40.388919 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:40.389419 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:40.389425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:40.389731 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:40.390159 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.390171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.390933 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.390937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.391711 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.391719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.392203 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.392206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.392365 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.392883 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:40.392897 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:40.394210 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:40.395556 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:40.395561 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:40.395572 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:40.395819 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:40.396123 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:40.396127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:40.396129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:40.396253 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:40.396627 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:40.396633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:40.396910 (+ 277us) client_negotiation.cc:769] Sending connection context
0923 14:38:40.397108 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 14:38:40.397392 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260923 14:38:40.399132 17021 heartbeater.cc:347] Connected to a master server at 127.2.218.62:45857
I20260923 14:38:40.399329 17021 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:40.399729 17021 heartbeater.cc:511] Master 127.2.218.62:45857 requested a full tablet report, sending...
I20260923 14:38:40.400636 16696 ts_manager.cc:194] Registered new tserver with Master: 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201)
I20260923 14:38:40.401185 16696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:58741
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:38:40.418996 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:45857
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:40.512619 17026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:40.512806 17026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:40.512840 17026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:40.516317 17026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:40.516364 17026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:40.516410 17026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:38:40.520720 17026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:40.521411 17026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:40.522214 17026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:40.532255 17034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:40.532282 17031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:40.532362 17032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:40.532575 17026 server_base.cc:1034] running on GCE node
I20260923 14:38:40.533082 17026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:40.533646 17026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:40.534825 17026 hybrid_clock.cc:648] HybridClock initialized: now 1790174320534817 us; error 40 us; skew 500 ppm
Sep 23 14:38:40 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:40.537323 17026 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:38:40.538004 17026 webserver.cc:460] Webserver started at http://127.2.218.3:34155/ using document root <none> and password file <none>
I20260923 14:38:40.538489 17026 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:40.538589 17026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:40.538805 17026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:40.540441 17026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "9270ced40adf72678564be703e17c137"
server_key_iv: "9e2d28af5caeec94c35310a44219153c"
server_key_version: "encryptionkey@0"
I20260923 14:38:40.540927 17026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "9270ced40adf72678564be703e17c137"
server_key_iv: "9e2d28af5caeec94c35310a44219153c"
server_key_version: "encryptionkey@0"
I20260923 14:38:40.544102 17026 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:38:40.546023 17041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:40.547010 17026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:40.547133 17026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "9270ced40adf72678564be703e17c137"
server_key_iv: "9e2d28af5caeec94c35310a44219153c"
server_key_version: "encryptionkey@0"
I20260923 14:38:40.547227 17026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:40.564308 17026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:40.566622 17026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:40.566751 17026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:40.567281 17026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:40.567966 17026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:40.568033 17026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:40.568089 17026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:40.568136 17026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:40.576120 17026 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:36747
I20260923 14:38:40.576144 17154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:36747 every 8 connection(s)
I20260923 14:38:40.576809 17026 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:38:40.583536 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 17026
I20260923 14:38:40.583667 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:38:40.583940 2920 external_mini_cluster.cc:1428] Setting key b85ae4fe20f5584daf4e945a143deb1d
Sep 23 14:38:40 dist-test-slave-zl47 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174320, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:40.588292 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.578279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:51659 (local address 127.2.218.62:45857)
0923 14:38:40.578422 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:40.578426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:40.579495 (+ 1069us) server_negotiation.cc:407] Connection header received
0923 14:38:40.580177 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:40.580180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:40.580222 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:40.580284 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:40.581247 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.581698 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.582692 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.582882 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.582952 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.585227 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:40.585231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:40.585233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:40.585257 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:40.586695 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:40.587114 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:40.587116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:40.587117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:40.587187 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:40.587469 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:40.587472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:40.587473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:40.587628 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:40.587705 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:40.588104 (+ 399us) server_negotiation.cc:299] Negotiation successful
0923 14:38:40.588190 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:38:40.588953 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.578631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:51659)
0923 14:38:40.579358 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:40.579393 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:40.580013 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:40.580422 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:40.580427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:40.580707 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:40.581125 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.581137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.581800 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.581802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.582590 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.582599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.582994 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.582997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.583117 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.583534 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:40.583556 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:40.585077 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:40.586785 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:40.586789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:40.586798 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:40.587001 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:40.587259 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:40.587261 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:40.587262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:40.587373 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:40.587728 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:40.587732 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:40.587909 (+ 177us) client_negotiation.cc:769] Sending connection context
0923 14:38:40.588055 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 14:38:40.588241 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":434,"threads_started":1}
I20260923 14:38:40.589942 17155 heartbeater.cc:347] Connected to a master server at 127.2.218.62:45857
I20260923 14:38:40.590165 17155 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:40.590524 17155 heartbeater.cc:511] Master 127.2.218.62:45857 requested a full tablet report, sending...
I20260923 14:38:40.591385 16696 ts_manager.cc:194] Registered new tserver with Master: 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747)
I20260923 14:38:40.591815 16696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:51659
I20260923 14:38:40.597365 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:38:40.604840 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.599007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.2.218.62:45857)
0923 14:38:40.599113 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:40.599116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:40.599264 (+ 148us) server_negotiation.cc:407] Connection header received
0923 14:38:40.599331 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:40.599333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:40.599365 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:40.599425 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:40.599928 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.600348 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.601070 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.601277 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.601376 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.602464 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:40.602467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:40.602468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:40.602487 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:40.603704 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:40.604008 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:40.604010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:40.604011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:40.604078 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:40.604258 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:40.604259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:40.604260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:40.604437 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:40.604530 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:40.604674 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 14:38:40.604746 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:38:40.607228 16696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:38:40.609726 16696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:38:40.621879 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.617184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.2.218.1:43471)
0923 14:38:40.617575 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:40.617586 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:40.617609 (+ 23us) server_negotiation.cc:407] Connection header received
0923 14:38:40.617678 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:40.617681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:40.617876 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:40.618010 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:40.618503 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.619174 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.620540 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.621258 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.621371 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.621501 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:40.621666 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 14:38:40.621761 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":147,"threads_started":1}
I20260923 14:38:40.621881 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.616941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:43471 (local address 127.0.0.1:33918)
0923 14:38:40.617258 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:40.617268 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:40.617327 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:40.618177 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:40.618182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:40.618199 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:40.618362 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.618369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.619292 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.619296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.620295 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.620303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.621391 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.621397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.621493 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.621526 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 14:38:40.621632 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 14:38:40.621739 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":41,"thread_start_us":88,"threads_started":1}
I20260923 14:38:40.621917 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.616554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:36747 (local address 127.0.0.1:48508)
0923 14:38:40.616897 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:40.616937 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:40.617057 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:40.618086 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:40.618125 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:40.618147 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:40.618309 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.618317 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.619694 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.619697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.620624 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.620632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.621508 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.621514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.621641 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.621686 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 14:38:40.621739 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 14:38:40.621796 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":53,"thread_start_us":72,"threads_started":1}
I20260923 14:38:40.622241 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.617226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40201 (local address 127.0.0.1:59094)
0923 14:38:40.617489 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:40.617501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:40.617568 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:40.618453 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:40.618455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:40.618472 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:40.618554 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.618559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.619735 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.619743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.620497 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.620508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.621608 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.621612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.621750 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.621772 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:38:40.621850 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:38:40.621913 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":53,"thread_start_us":117,"threads_started":1}
I20260923 14:38:40.622651 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.616679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.2.218.3:36747)
0923 14:38:40.617612 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:40.617624 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:40.617643 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:38:40.617729 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:40.617734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:40.617916 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:40.618456 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:40.618579 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.619559 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.620735 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.621395 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.622279 (+ 884us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.622431 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:40.622476 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 14:38:40.622545 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":128,"threads_started":1}
I20260923 14:38:40.622670 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.617655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.2.218.2:40201)
0923 14:38:40.618044 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:40.618057 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:40.618076 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:38:40.618164 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:40.618168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:40.618329 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:40.618458 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:40.618663 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.619609 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.620639 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.621298 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.621425 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.621548 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:40.622498 (+ 950us) server_negotiation.cc:299] Negotiation successful
0923 14:38:40.622570 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":125,"threads_started":1}
I20260923 14:38:40.624590 16821 tablet_service.cc:1467] Processing CreateTablet for tablet 7b94f3f41b3f451db9ef7a485a5e01c5 (DEFAULT_TABLE table=test-table [id=1e0797461a4c41ac8c84bc4113bf805c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:40.624876 17089 tablet_service.cc:1467] Processing CreateTablet for tablet 7b94f3f41b3f451db9ef7a485a5e01c5 (DEFAULT_TABLE table=test-table [id=1e0797461a4c41ac8c84bc4113bf805c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:40.624877 16955 tablet_service.cc:1467] Processing CreateTablet for tablet 7b94f3f41b3f451db9ef7a485a5e01c5 (DEFAULT_TABLE table=test-table [id=1e0797461a4c41ac8c84bc4113bf805c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:40.625402 16821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b94f3f41b3f451db9ef7a485a5e01c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:40.625722 16955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b94f3f41b3f451db9ef7a485a5e01c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:40.625730 17089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b94f3f41b3f451db9ef7a485a5e01c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:40.630589 17172 tablet_bootstrap.cc:492] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Bootstrap starting.
I20260923 14:38:40.631800 17174 tablet_bootstrap.cc:492] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: Bootstrap starting.
I20260923 14:38:40.633659 17172 tablet_bootstrap.cc:654] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:40.634014 17174 tablet_bootstrap.cc:654] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:40.634490 17172 log.cc:824] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:40.634755 17174 log.cc:824] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:40.635131 17173 tablet_bootstrap.cc:492] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Bootstrap starting.
I20260923 14:38:40.636397 17174 tablet_bootstrap.cc:492] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: No bootstrap required, opened a new log
I20260923 14:38:40.636426 17172 tablet_bootstrap.cc:492] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: No bootstrap required, opened a new log
I20260923 14:38:40.636581 17174 ts_tablet_manager.cc:1397] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:38:40.636667 17172 ts_tablet_manager.cc:1397] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:38:40.637306 17173 tablet_bootstrap.cc:654] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:40.637971 17173 log.cc:824] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:40.639700 17173 tablet_bootstrap.cc:492] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: No bootstrap required, opened a new log
I20260923 14:38:40.639868 17173 ts_tablet_manager.cc:1397] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:38:40.639740 17172 raft_consensus.cc:348] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:40.640030 17172 raft_consensus.cc:374] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:40.640081 17172 raft_consensus.cc:729] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995089815202416fb2af2712aae7ef4b, State: Initialized, Role: FOLLOWER
I20260923 14:38:40.639951 17174 raft_consensus.cc:348] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:40.640188 17174 raft_consensus.cc:374] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:40.640247 17174 raft_consensus.cc:729] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9235ee37ce7d4d1788898e1bed052ca8, State: Initialized, Role: FOLLOWER
I20260923 14:38:40.640529 17172 consensus_queue.cc:260] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:40.640615 17174 consensus_queue.cc:260] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:40.641073 16887 heartbeater.cc:503] Master 127.2.218.62:45857 was elected leader, sending a full tablet report...
I20260923 14:38:40.641376 17172 ts_tablet_manager.cc:1428] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:38:40.641491 17174 ts_tablet_manager.cc:1428] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:38:40.641795 17021 heartbeater.cc:503] Master 127.2.218.62:45857 was elected leader, sending a full tablet report...
I20260923 14:38:40.643257 17173 raft_consensus.cc:348] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:40.643513 17173 raft_consensus.cc:374] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:40.643568 17173 raft_consensus.cc:729] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e42dfe32eb64f61ac30a83b88bd8ce7, State: Initialized, Role: FOLLOWER
I20260923 14:38:40.643949 17173 consensus_queue.cc:260] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:40.644599 17155 heartbeater.cc:503] Master 127.2.218.62:45857 was elected leader, sending a full tablet report...
I20260923 14:38:40.645234 17173 ts_tablet_manager.cc:1428] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260923 14:38:40.690500 16888 tablet.cc:2369] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:38:40.827653 17156 tablet.cc:2369] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:38:40.886631 17022 tablet.cc:2369] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:38:40.901849 17180 raft_consensus.cc:482] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:38:40.902010 17180 raft_consensus.cc:504] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:40.903090 17180 leader_election.cc:288] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201), 995089815202416fb2af2712aae7ef4b (127.2.218.1:43471)
I20260923 14:38:40.906431 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.903275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:43471 (local address 127.2.218.3:59017)
0923 14:38:40.903388 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:40.903403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:40.903505 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:40.903776 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:40.903779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:40.903805 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:40.903920 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.903925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.904609 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.904614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.905297 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.905305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.905937 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.905941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.906061 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.906082 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 14:38:40.906207 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 14:38:40.906292 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 14:38:40.906447 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.903408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:59017 (local address 127.2.218.1:43471)
0923 14:38:40.903549 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:40.903553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:40.903567 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:38:40.903626 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:40.903629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:40.903667 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:40.903765 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:40.904032 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.904498 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.905422 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.905810 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.905936 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.906008 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:40.906248 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 14:38:40.906336 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 14:38:40.906814 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.903409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:57931 (local address 127.2.218.2:40201)
0923 14:38:40.903550 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:40.903556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:40.903953 (+ 397us) server_negotiation.cc:407] Connection header received
0923 14:38:40.904072 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:40.904075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:40.904122 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:40.904195 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:40.904429 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.904852 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.905756 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.906223 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.906356 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.906413 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:40.906623 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 14:38:40.906694 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:38:40.907158 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.903275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40201 (local address 127.2.218.3:57931)
0923 14:38:40.903871 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:40.903880 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:40.903979 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:40.904203 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:40.904206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:40.904220 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:40.904331 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.904337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.904954 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.904959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.905620 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:40.905627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.906360 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.906364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:40.906466 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.906487 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 14:38:40.906974 (+ 487us) client_negotiation.cc:240] Negotiation successful
0923 14:38:40.907039 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":296,"thread_start_us":118,"threads_started":1}
I20260923 14:38:40.907131 16841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b94f3f41b3f451db9ef7a485a5e01c5" candidate_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995089815202416fb2af2712aae7ef4b" is_pre_election: true
I20260923 14:38:40.907459 16841 raft_consensus.cc:2454] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e42dfe32eb64f61ac30a83b88bd8ce7 in term 0.
I20260923 14:38:40.907792 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b94f3f41b3f451db9ef7a485a5e01c5" candidate_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9235ee37ce7d4d1788898e1bed052ca8" is_pre_election: true
I20260923 14:38:40.907944 17044 leader_election.cc:302] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e42dfe32eb64f61ac30a83b88bd8ce7, 995089815202416fb2af2712aae7ef4b; no voters:
I20260923 14:38:40.908121 16975 raft_consensus.cc:2454] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e42dfe32eb64f61ac30a83b88bd8ce7 in term 0.
I20260923 14:38:40.908248 17180 raft_consensus.cc:2781] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:38:40.908358 17180 raft_consensus.cc:482] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:38:40.908445 17180 raft_consensus.cc:3037] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:40.909512 17180 raft_consensus.cc:504] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:40.909811 17180 leader_election.cc:288] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [CANDIDATE]: Term 1 election: Requested vote from peers 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201), 995089815202416fb2af2712aae7ef4b (127.2.218.1:43471)
I20260923 14:38:40.910130 16975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b94f3f41b3f451db9ef7a485a5e01c5" candidate_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9235ee37ce7d4d1788898e1bed052ca8"
I20260923 14:38:40.910195 16841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b94f3f41b3f451db9ef7a485a5e01c5" candidate_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995089815202416fb2af2712aae7ef4b"
I20260923 14:38:40.910296 16975 raft_consensus.cc:3037] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:40.910316 16841 raft_consensus.cc:3037] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:40.911219 16841 raft_consensus.cc:2454] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e42dfe32eb64f61ac30a83b88bd8ce7 in term 1.
I20260923 14:38:40.911226 16975 raft_consensus.cc:2454] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e42dfe32eb64f61ac30a83b88bd8ce7 in term 1.
I20260923 14:38:40.911466 17044 leader_election.cc:302] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e42dfe32eb64f61ac30a83b88bd8ce7, 995089815202416fb2af2712aae7ef4b; no voters:
I20260923 14:38:40.911626 17180 raft_consensus.cc:2781] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:38:40.911830 17180 raft_consensus.cc:686] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 1 LEADER]: Becoming Leader. State: Replica: 7e42dfe32eb64f61ac30a83b88bd8ce7, State: Running, Role: LEADER
I20260923 14:38:40.912120 17180 consensus_queue.cc:237] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:40.915186 16694 catalog_manager.cc:5495] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3). New cstate: current_term: 1 leader_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } health_report { overall_health: UNKNOWN } } }
I20260923 14:38:40.978685 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:40.975883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48518 (local address 127.2.218.3:36747)
0923 14:38:40.975996 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:40.976000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:40.976045 (+ 45us) server_negotiation.cc:407] Connection header received
0923 14:38:40.976169 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:40.976171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:40.976212 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:40.976284 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:38:40.976474 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.976894 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.977602 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:40.977769 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:40.977832 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:40.978052 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:38:40.978389 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:38:40.978464 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:40.978553 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 14:38:40.978590 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:38:40.983400 16841 raft_consensus.cc:1261] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Refusing update from remote peer 7e42dfe32eb64f61ac30a83b88bd8ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:38:40.983428 16975 raft_consensus.cc:1261] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8 [term 1 FOLLOWER]: Refusing update from remote peer 7e42dfe32eb64f61ac30a83b88bd8ce7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:38:40.984027 17180 consensus_queue.cc:1035] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:38:40.984192 17182 consensus_queue.cc:1035] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:38:40.992976 17186 mvcc.cc:205] Tried to move back new op lower bound from 7332554018742018048 to 7332554018458697728. Current Snapshot: MvccSnapshot[applied={T|T < 7332554018742018048 or (T in {7332554018742018048})}]
I20260923 14:38:40.993122 17185 mvcc.cc:205] Tried to move back new op lower bound from 7332554018742018048 to 7332554018458697728. Current Snapshot: MvccSnapshot[applied={T|T < 7332554018742018048 or (T in {7332554018742018048})}]
I20260923 14:38:40.996625 16696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:38:40.996732 16696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:38:40.998792 16696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Sending DeleteTablet for 3 replicas of tablet 7b94f3f41b3f451db9ef7a485a5e01c5
I20260923 14:38:40.999464 16821 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b94f3f41b3f451db9ef7a485a5e01c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:40 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:33918
I20260923 14:38:40.999529 16955 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b94f3f41b3f451db9ef7a485a5e01c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:40 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:59094
I20260923 14:38:40.999655 17089 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b94f3f41b3f451db9ef7a485a5e01c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:40 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:48508
I20260923 14:38:40.999894 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 16758
I20260923 14:38:41.000065 17196 tablet_replica.cc:330] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: stopping tablet replica
I20260923 14:38:41.000346 17196 raft_consensus.cc:2229] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:38:41.000552 17194 tablet_replica.cc:330] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: stopping tablet replica
I20260923 14:38:41.000628 17196 raft_consensus.cc:2258] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:38:41.001151 17194 raft_consensus.cc:2229] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:38:41.001449 17194 raft_consensus.cc:2258] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:38:41.001796 17196 ts_tablet_manager.cc:1905] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:38:41.002732 17194 ts_tablet_manager.cc:1905] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:38:41.004812 17196 ts_tablet_manager.cc:1918] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:38:41.004902 17196 log.cc:1196] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/7b94f3f41b3f451db9ef7a485a5e01c5
I20260923 14:38:41.005129 17194 ts_tablet_manager.cc:1918] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:38:41.005199 17196 ts_tablet_manager.cc:1939] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Deleting consensus metadata
I20260923 14:38:41.005290 17194 log.cc:1196] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/7b94f3f41b3f451db9ef7a485a5e01c5
I20260923 14:38:41.005595 17194 ts_tablet_manager.cc:1939] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 9235ee37ce7d4d1788898e1bed052ca8: Deleting consensus metadata
I20260923 14:38:41.006191 16683 catalog_manager.cc:4841] TS 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747): tablet 7b94f3f41b3f451db9ef7a485a5e01c5 (table test-table [id=1e0797461a4c41ac8c84bc4113bf805c]) successfully deleted
I20260923 14:38:41.006757 16683 catalog_manager.cc:4841] TS 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201): tablet 7b94f3f41b3f451db9ef7a485a5e01c5 (table test-table [id=1e0797461a4c41ac8c84bc4113bf805c]) successfully deleted
W20260923 14:38:41.007565 16682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:43471 (error 108)
I20260923 14:38:41.008273 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 16892
I20260923 14:38:41.008417 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.008164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:43471 (local address 127.0.0.1:33924)
0923 14:38:41.008297 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:41.008356 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:43471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260923 14:38:41.009167 16682 catalog_manager.cc:4584] TS 995089815202416fb2af2712aae7ef4b (127.2.218.1:43471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b94f3f41b3f451db9ef7a485a5e01c5: Network error: Client connection negotiation failed: client connection to 127.2.218.1:43471: connect: Connection refused (error 111)
I20260923 14:38:41.013370 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 17026
I20260923 14:38:41.017379 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 16664
I20260923 14:38:41.022897 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:45857
--webserver_interface=127.2.218.62
--webserver_port=38531
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:45857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:41.114859 17198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:41.115061 17198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:41.115098 17198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:41.118548 17198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:38:41.118585 17198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:41.118602 17198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:41.118615 17198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:38:41.118669 17198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:38:41.122808 17198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:45857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:45857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=38531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:41.123576 17198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:41.124385 17198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:41.136024 17207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:41.136057 17204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:41.136092 17198 server_base.cc:1034] running on GCE node
W20260923 14:38:41.136029 17205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:41.136698 17198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:41.137599 17198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:41.138778 17198 hybrid_clock.cc:648] HybridClock initialized: now 1790174321138767 us; error 28 us; skew 500 ppm
Sep 23 14:38:41 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:41.144450 17198 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:38:41.145190 17198 webserver.cc:460] Webserver started at http://127.2.218.62:38531/ using document root <none> and password file <none>
I20260923 14:38:41.145658 17198 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:41.145745 17198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:41.148775 17198 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:41.150595 17214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:41.151616 17198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:41.151739 17198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "96e737264c3b40e3bcc193b5eeacbb4a"
format_stamp: "Formatted at 2026-09-23 14:38:39 on dist-test-slave-zl47"
server_key: "ccbc395566012442c2019b372b9874db"
server_key_iv: "6d4ba78025a56d3309fcfbc9868ac80f"
server_key_version: "encryptionkey@0"
I20260923 14:38:41.152206 17198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:41.161571 17198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:41.163923 17198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:41.164108 17198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:41.170034 17198 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:45857
I20260923 14:38:41.170295 17266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:45857 every 8 connection(s)
I20260923 14:38:41.171031 17198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:38:41.174914 17267 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:38:41.176462 17267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Bootstrap starting.
I20260923 14:38:41.177551 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 17198
I20260923 14:38:41.177928 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:43471
--local_ip_for_outbound_sockets=127.2.218.1
--tserver_master_addrs=127.2.218.62:45857
--webserver_port=45935
--webserver_interface=127.2.218.1
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:38:41.192216 17267 log.cc:824] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:41.199499 17267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:38:41.199812 17267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Bootstrap complete.
I20260923 14:38:41.203318 17267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:41.203668 17267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e737264c3b40e3bcc193b5eeacbb4a, State: Initialized, Role: FOLLOWER
I20260923 14:38:41.204120 17267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:41.204223 17267 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:38:41.204288 17267 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:38:41.204389 17267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:38:41.205999 17267 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:41.206301 17267 leader_election.cc:302] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96e737264c3b40e3bcc193b5eeacbb4a; no voters:
I20260923 14:38:41.206583 17267 leader_election.cc:288] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 14:38:41.206648 17271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 2 FOLLOWER]: Leader election won for term 2
I20260923 14:38:41.206835 17271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 2 LEADER]: Becoming Leader. State: Replica: 96e737264c3b40e3bcc193b5eeacbb4a, State: Running, Role: LEADER
I20260923 14:38:41.207090 17271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:41.207619 17267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:38:41.207849 17272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } } }
I20260923 14:38:41.207967 17272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:41.208250 17273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96e737264c3b40e3bcc193b5eeacbb4a. Latest consensus state: current_term: 2 leader_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } } }
I20260923 14:38:41.208959 17275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:38:41.208968 17273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:41.212960 17275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:38:41.213500 17275 catalog_manager.cc:1259] Loaded cluster ID: d45d59d8aee54e5e9aaf34d0ed3e002d
I20260923 14:38:41.213622 17275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:38:41.214927 17275 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 14:38:41.219123 17275 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 14:38:41.223690 17275 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:38:41.224347 17275 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Loaded TSK: 0
I20260923 14:38:41.225487 17275 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:38:41.278568 17269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:41.278743 17269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:41.278779 17269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:41.282235 17269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:41.282280 17269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:41.282364 17269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:38:41.286603 17269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:43471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=45935
--tserver_master_addrs=127.2.218.62:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:41.287334 17269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:41.288208 17269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:41.297245 17295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:41.297262 17294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:41.297320 17297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:41.297571 17269 server_base.cc:1034] running on GCE node
I20260923 14:38:41.297973 17269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:41.298590 17269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:41.299772 17269 hybrid_clock.cc:648] HybridClock initialized: now 1790174321299752 us; error 35 us; skew 500 ppm
Sep 23 14:38:41 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:41.302444 17269 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:38:41.303288 17269 webserver.cc:460] Webserver started at http://127.2.218.1:45935/ using document root <none> and password file <none>
I20260923 14:38:41.303766 17269 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:41.303874 17269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:41.307220 17269 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:38:41.309195 17304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:41.310338 17269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:41.310459 17269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "995089815202416fb2af2712aae7ef4b"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "4789cb2bfded6d3d5d30fdbd5a6fe63a"
server_key_iv: "2a196b8eb4e31c7a65713df5ee403aa4"
server_key_version: "encryptionkey@0"
I20260923 14:38:41.310881 17269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:41.323947 17269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:41.326467 17269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:41.326642 17269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:41.327158 17269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:41.328080 17311 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 14:38:41.331343 17269 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 14:38:41.331403 17269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260923 14:38:41.331459 17269 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 14:38:41.332894 17269 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 14:38:41.332939 17269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 14:38:41.333161 17311 tablet_bootstrap.cc:492] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Bootstrap starting.
I20260923 14:38:41.343101 17269 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:43471
I20260923 14:38:41.343192 17418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:43471 every 8 connection(s)
I20260923 14:38:41.343901 17269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:38:41.348474 17311 log.cc:824] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Log is configured to *not* fsync() on all Append() calls
Sep 23 14:38:41 dist-test-slave-zl47 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:41.353633 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 17269
I20260923 14:38:41.354072 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:40201
--local_ip_for_outbound_sockets=127.2.218.2
--tserver_master_addrs=127.2.218.62:45857
--webserver_port=44261
--webserver_interface=127.2.218.2
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:38:41.355536 17311 tablet_bootstrap.cc:492] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:38:41.355893 17311 tablet_bootstrap.cc:492] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Bootstrap complete.
I20260923 14:38:41.357007 17311 ts_tablet_manager.cc:1397] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20260923 14:38:41.359352 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.345683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:35163)
0923 14:38:41.346343 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:41.346389 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:41.347043 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:41.348095 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:41.348100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:41.348422 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:41.348852 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.348865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.350039 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.350042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.351177 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.351188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.351805 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.351809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.351972 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.352445 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:41.352463 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:41.354049 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:41.356306 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:41.356314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:41.356327 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:41.356648 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:41.357028 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:41.357031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:41.357033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:41.357186 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:41.357851 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:41.357857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:41.358090 (+ 233us) client_negotiation.cc:769] Sending connection context
0923 14:38:41.358285 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 14:38:41.358567 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":131,"threads_started":1}
I20260923 14:38:41.359542 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.345760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:35163 (local address 127.2.218.62:45857)
0923 14:38:41.346126 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.346142 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.346522 (+ 380us) server_negotiation.cc:407] Connection header received
0923 14:38:41.347402 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.347415 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.347745 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.348060 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:41.349018 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.349891 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.351287 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.351664 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.351767 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.354419 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:41.354424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:41.354445 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:41.354481 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:41.356160 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:41.356769 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:41.356777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:41.356782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:41.356922 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:41.357303 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:41.357308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:41.357310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:41.357724 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:41.357840 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.358428 (+ 588us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.358722 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":107,"threads_started":1}
I20260923 14:38:41.359339 17311 raft_consensus.cc:348] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.359831 17311 raft_consensus.cc:729] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995089815202416fb2af2712aae7ef4b, State: Initialized, Role: FOLLOWER
I20260923 14:38:41.360250 17311 consensus_queue.cc:260] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.361023 17311 ts_tablet_manager.cc:1428] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260923 14:38:41.361510 17419 heartbeater.cc:347] Connected to a master server at 127.2.218.62:45857
I20260923 14:38:41.361721 17419 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:41.362218 17419 heartbeater.cc:511] Master 127.2.218.62:45857 requested a full tablet report, sending...
I20260923 14:38:41.364566 17231 ts_manager.cc:194] Registered new tserver with Master: 995089815202416fb2af2712aae7ef4b (127.2.218.1:43471)
I20260923 14:38:41.367386 17231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:35163
I20260923 14:38:41.369401 17419 heartbeater.cc:503] Master 127.2.218.62:45857 was elected leader, sending a full tablet report...
Sep 23 14:38:41 dist-test-slave-zl47 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:38:41.376578 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.366848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:43471 (local address 127.0.0.1:33934)
0923 14:38:41.367429 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:41.367465 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:41.367568 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:41.368117 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:41.368122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:41.368453 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:41.368563 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.368569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.369794 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.369797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.370685 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.370693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.371263 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.371267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.371384 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.371848 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:41.371859 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:41.373194 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:41.374967 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:41.374976 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:41.374980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:41.375243 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:41.375565 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:41.375569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:41.375572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:41.375658 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:41.376053 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:41.376058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:41.376161 (+ 103us) client_negotiation.cc:769] Sending connection context
0923 14:38:41.376277 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 14:38:41.376439 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":80,"threads_started":1}
I20260923 14:38:41.376605 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.366936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.2.218.1:43471)
0923 14:38:41.367222 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.367230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.367679 (+ 449us) server_negotiation.cc:407] Connection header received
0923 14:38:41.367732 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.367736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.367891 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.368008 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:41.368703 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.369577 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.370771 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.371133 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.371260 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.373348 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:41.373361 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:41.373371 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:41.373400 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:41.374835 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:41.375369 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:41.375374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:41.375377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:41.375461 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:41.375747 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:41.375751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:41.375754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:41.375945 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:41.376071 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.376356 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.376484 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":95,"threads_started":1}
I20260923 14:38:41.377502 17353 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b94f3f41b3f451db9ef7a485a5e01c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:40 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:33934
I20260923 14:38:41.378260 17428 tablet_replica.cc:330] stopping tablet replica
I20260923 14:38:41.378532 17428 raft_consensus.cc:2229] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:38:41.378736 17428 raft_consensus.cc:2258] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:38:41.379614 17428 ts_tablet_manager.cc:1905] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:38:41.383283 17428 ts_tablet_manager.cc:1918] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:38:41.383373 17428 log.cc:1196] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/7b94f3f41b3f451db9ef7a485a5e01c5
I20260923 14:38:41.383637 17428 ts_tablet_manager.cc:1939] T 7b94f3f41b3f451db9ef7a485a5e01c5 P 995089815202416fb2af2712aae7ef4b: Deleting consensus metadata
I20260923 14:38:41.385116 17217 catalog_manager.cc:4841] TS 995089815202416fb2af2712aae7ef4b (127.2.218.1:43471): tablet 7b94f3f41b3f451db9ef7a485a5e01c5 (table test-table [id=1e0797461a4c41ac8c84bc4113bf805c]) successfully deleted
W20260923 14:38:41.454540 17424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:41.454762 17424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:41.454824 17424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:41.458439 17424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:41.458484 17424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:41.458539 17424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:38:41.462812 17424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:40201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=44261
--tserver_master_addrs=127.2.218.62:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:41.463564 17424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:41.464336 17424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:41.471992 17433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:41.472002 17434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:41.472133 17424 server_base.cc:1034] running on GCE node
W20260923 14:38:41.471998 17436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:41.472570 17424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:41.473066 17424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:41.474176 17424 hybrid_clock.cc:648] HybridClock initialized: now 1790174321474140 us; error 40 us; skew 500 ppm
Sep 23 14:38:41 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:41.476432 17424 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:38:41.477084 17424 webserver.cc:460] Webserver started at http://127.2.218.2:44261/ using document root <none> and password file <none>
I20260923 14:38:41.477522 17424 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:41.477607 17424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:41.480552 17424 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:41.482172 17443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:41.483150 17424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:41.483269 17424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "9235ee37ce7d4d1788898e1bed052ca8"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "efa63c4afd30dd6ceea517bb143d86e7"
server_key_iv: "3ed740ffd6d3c2b9cc7da4fe2ebed666"
server_key_version: "encryptionkey@0"
I20260923 14:38:41.483750 17424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:41.509266 17424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:41.511444 17424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:41.511616 17424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:41.512099 17424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:41.512722 17424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:41.512784 17424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:41.512840 17424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:41.512882 17424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:41.520587 17424 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:40201
I20260923 14:38:41.520625 17556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:40201 every 8 connection(s)
I20260923 14:38:41.521341 17424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
Sep 23 14:38:41 dist-test-slave-zl47 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:41.531025 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 17424
I20260923 14:38:41.531399 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:36747
--local_ip_for_outbound_sockets=127.2.218.3
--tserver_master_addrs=127.2.218.62:45857
--webserver_port=34155
--webserver_interface=127.2.218.3
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:38:41.531853 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.522761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:58615 (local address 127.2.218.62:45857)
0923 14:38:41.522885 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.522889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.523735 (+ 846us) server_negotiation.cc:407] Connection header received
0923 14:38:41.524458 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.524460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.524510 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.524572 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:41.525440 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.525874 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.526845 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.527037 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.527100 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.529024 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:41.529027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:41.529028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:41.529049 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:41.530125 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:41.530508 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:41.530510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:41.530511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:41.530575 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:41.530889 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:41.530892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:41.530893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:41.531113 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:41.531211 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.531628 (+ 417us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.531720 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:38:41.532778 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.523075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:58615)
0923 14:38:41.523583 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:41.523614 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:41.524286 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:41.524704 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:41.524709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:41.524970 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:41.525335 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.525346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.525972 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.525974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.526748 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.526756 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.527139 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.527142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.527252 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.527722 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:41.527734 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:41.528892 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:41.530212 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:41.530216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:41.530224 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:41.530432 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:41.530652 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:41.530653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:41.530655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:41.530759 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:41.531207 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:41.531211 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:41.531376 (+ 165us) client_negotiation.cc:769] Sending connection context
0923 14:38:41.532008 (+ 632us) client_negotiation.cc:240] Negotiation successful
0923 14:38:41.532233 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":82,"threads_started":1}
I20260923 14:38:41.533735 17557 heartbeater.cc:347] Connected to a master server at 127.2.218.62:45857
I20260923 14:38:41.533934 17557 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:41.534353 17557 heartbeater.cc:511] Master 127.2.218.62:45857 requested a full tablet report, sending...
I20260923 14:38:41.535181 17231 ts_manager.cc:194] Registered new tserver with Master: 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201)
I20260923 14:38:41.535653 17231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:58615
W20260923 14:38:41.623250 17560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:41.623457 17560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:41.623521 17560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:41.627107 17560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:41.627171 17560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:41.627264 17560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:38:41.631585 17560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:36747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=34155
--tserver_master_addrs=127.2.218.62:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:41.632388 17560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:41.633176 17560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:41.641013 17565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:41.641028 17566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:41.641084 17560 server_base.cc:1034] running on GCE node
W20260923 14:38:41.641028 17568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:41.641561 17560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:41.642005 17560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:41.643146 17560 hybrid_clock.cc:648] HybridClock initialized: now 1790174321643126 us; error 36 us; skew 500 ppm
Sep 23 14:38:41 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:41.645335 17560 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:38:41.646024 17560 webserver.cc:460] Webserver started at http://127.2.218.3:34155/ using document root <none> and password file <none>
I20260923 14:38:41.646543 17560 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:41.646605 17560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:41.649541 17560 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:38:41.651203 17575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:41.652194 17560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:38:41.652313 17560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "9270ced40adf72678564be703e17c137"
server_key_iv: "9e2d28af5caeec94c35310a44219153c"
server_key_version: "encryptionkey@0"
I20260923 14:38:41.652753 17560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:41.676314 17560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:41.678542 17560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:41.678705 17560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:41.679193 17560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:41.679862 17560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:41.679926 17560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:41.679984 17560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:41.680033 17560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:41.687974 17560 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:36747
I20260923 14:38:41.688019 17688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:36747 every 8 connection(s)
I20260923 14:38:41.688791 17560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
Sep 23 14:38:41 dist-test-slave-zl47 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174321, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:41.697026 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 17560
I20260923 14:38:41.701027 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.690387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:46357 (local address 127.2.218.62:45857)
0923 14:38:41.690507 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.690510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.691320 (+ 810us) server_negotiation.cc:407] Connection header received
0923 14:38:41.692100 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.692104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.692162 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.692263 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:41.693386 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.693855 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.694917 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.695106 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.695176 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.697291 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:41.697295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:41.697298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:41.697322 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:41.699019 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:41.699626 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:41.699629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:41.699631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:41.699725 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:41.700026 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:41.700029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:41.700030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:41.700224 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:41.700296 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.700836 (+ 540us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.700917 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:38:41.701840 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.690682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:46357)
0923 14:38:41.691181 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:41.691213 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:41.691903 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:41.692421 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:41.692428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:41.692735 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:41.693232 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.693250 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.693984 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.693986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.694820 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.694828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.695219 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.695222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.695346 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.695842 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:41.695856 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:41.697147 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:41.699145 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:41.699151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:41.699165 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:41.699533 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:41.699817 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:41.699819 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:41.699820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:41.699940 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:41.700352 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:41.700358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:41.700608 (+ 250us) client_negotiation.cc:769] Sending connection context
0923 14:38:41.700803 (+ 195us) client_negotiation.cc:240] Negotiation successful
0923 14:38:41.701084 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":89,"threads_started":1}
I20260923 14:38:41.702930 17689 heartbeater.cc:347] Connected to a master server at 127.2.218.62:45857
I20260923 14:38:41.703135 17689 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:41.703534 17689 heartbeater.cc:511] Master 127.2.218.62:45857 requested a full tablet report, sending...
I20260923 14:38:41.704489 17231 ts_manager.cc:194] Registered new tserver with Master: 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747)
I20260923 14:38:41.705233 17231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:46357
I20260923 14:38:41.705920 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.698026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42888 (local address 127.2.218.62:45857)
0923 14:38:41.698346 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.698349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.698361 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:38:41.698392 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.698394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.698429 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.698539 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:41.699424 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.700077 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.701187 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.701474 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.701579 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.702804 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:41.702807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:41.702809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:41.702827 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:41.704349 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:41.704791 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:41.704793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:41.704795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:41.704877 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:41.705130 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:41.705133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:41.705134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:41.705283 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:41.705538 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.705670 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.705778 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260923 14:38:41.707425 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:38:41.711572 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.708028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.2.218.62:45857)
0923 14:38:41.708120 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.708124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.708343 (+ 219us) server_negotiation.cc:407] Connection header received
0923 14:38:41.708499 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.708501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.708566 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.708631 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:38:41.708917 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.709483 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.710240 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.710492 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.710580 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.711066 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:38:41.711320 (+ 254us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:38:41.711395 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.711470 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.711506 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 14:38:41.712138 17231 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:38:41.715164 17231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:38:41.721971 17353 tablet_service.cc:1467] Processing CreateTablet for tablet 514539aadb53489f8cde29b4646a86c7 (DEFAULT_TABLE table=test-table [id=960bf78502b94fccabd5feb2931e3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:41.722587 17353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514539aadb53489f8cde29b4646a86c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:41.725335 17311 tablet_bootstrap.cc:492] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Bootstrap starting.
I20260923 14:38:41.725528 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.720918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:36747 (local address 127.0.0.1:48532)
0923 14:38:41.721064 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:41.721077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:41.721166 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:41.721937 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:41.721939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:41.721955 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:41.722051 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.722057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.723376 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.723380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.724387 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.724398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.725180 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.725183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.725259 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.725292 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 14:38:41.725363 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 14:38:41.725429 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 14:38:41.725574 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.721010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.2.218.3:36747)
0923 14:38:41.721482 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.721490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.721528 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:38:41.721586 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.721591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.721760 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.721950 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:41.722181 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.723116 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.724505 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.725072 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.725177 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.725286 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.725397 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.725467 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":124,"threads_started":1}
I20260923 14:38:41.726202 17311 tablet_bootstrap.cc:654] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:41.726260 17696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.721224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40201 (local address 127.0.0.1:59096)
0923 14:38:41.721692 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:41.721707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:41.721787 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:41.722497 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:41.722501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:41.722518 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:41.722595 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.722603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.723811 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.723814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.724675 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.724684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.725835 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.725839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.725916 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.725938 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:38:41.726009 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 14:38:41.726081 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":78,"threads_started":1}
I20260923 14:38:41.726903 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.721610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.2.218.2:40201)
0923 14:38:41.721985 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.721995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.722013 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:38:41.722089 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.722159 (+ 70us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.722357 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.722487 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:41.722714 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.723679 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.724800 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.725540 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.725643 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.725771 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.726494 (+ 723us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.726758 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":96,"threads_started":1}
I20260923 14:38:41.728204 17311 tablet_bootstrap.cc:492] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: No bootstrap required, opened a new log
I20260923 14:38:41.728281 17311 ts_tablet_manager.cc:1397] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260923 14:38:41.727968 17623 tablet_service.cc:1467] Processing CreateTablet for tablet 514539aadb53489f8cde29b4646a86c7 (DEFAULT_TABLE table=test-table [id=960bf78502b94fccabd5feb2931e3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:41.728662 17311 raft_consensus.cc:348] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.728765 17311 raft_consensus.cc:374] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:41.728797 17311 raft_consensus.cc:729] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995089815202416fb2af2712aae7ef4b, State: Initialized, Role: FOLLOWER
I20260923 14:38:41.728868 17623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514539aadb53489f8cde29b4646a86c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:41.728920 17311 consensus_queue.cc:260] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.729140 17311 ts_tablet_manager.cc:1428] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 14:38:41.729328 17491 tablet_service.cc:1467] Processing CreateTablet for tablet 514539aadb53489f8cde29b4646a86c7 (DEFAULT_TABLE table=test-table [id=960bf78502b94fccabd5feb2931e3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:41.730049 17491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514539aadb53489f8cde29b4646a86c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:41.734352 17699 tablet_bootstrap.cc:492] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Bootstrap starting.
I20260923 14:38:41.735975 17700 tablet_bootstrap.cc:492] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: Bootstrap starting.
I20260923 14:38:41.736600 17699 tablet_bootstrap.cc:654] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:41.737390 17699 log.cc:824] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:41.738421 17700 tablet_bootstrap.cc:654] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:41.739159 17700 log.cc:824] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:41.739271 17699 tablet_bootstrap.cc:492] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: No bootstrap required, opened a new log
I20260923 14:38:41.739446 17699 ts_tablet_manager.cc:1397] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:38:41.740826 17700 tablet_bootstrap.cc:492] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: No bootstrap required, opened a new log
I20260923 14:38:41.741000 17700 ts_tablet_manager.cc:1397] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260923 14:38:41.742897 17699 raft_consensus.cc:348] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.743140 17699 raft_consensus.cc:374] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:41.743197 17699 raft_consensus.cc:729] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e42dfe32eb64f61ac30a83b88bd8ce7, State: Initialized, Role: FOLLOWER
I20260923 14:38:41.743588 17699 consensus_queue.cc:260] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.744138 17689 heartbeater.cc:503] Master 127.2.218.62:45857 was elected leader, sending a full tablet report...
I20260923 14:38:41.744500 17699 ts_tablet_manager.cc:1428] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 14:38:41.744422 17700 raft_consensus.cc:348] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.744668 17700 raft_consensus.cc:374] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:41.744755 17700 raft_consensus.cc:729] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9235ee37ce7d4d1788898e1bed052ca8, State: Initialized, Role: FOLLOWER
I20260923 14:38:41.745132 17700 consensus_queue.cc:260] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.745683 17557 heartbeater.cc:503] Master 127.2.218.62:45857 was elected leader, sending a full tablet report...
I20260923 14:38:41.745944 17700 ts_tablet_manager.cc:1428] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
W20260923 14:38:41.772091 17558 tablet.cc:2369] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:38:41.844947 17420 tablet.cc:2369] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:38:41.914938 17704 raft_consensus.cc:482] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:38:41.915138 17704 raft_consensus.cc:504] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.916254 17704 leader_election.cc:288] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747), 995089815202416fb2af2712aae7ef4b (127.2.218.1:43471)
I20260923 14:38:41.919451 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.916476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:36747 (local address 127.2.218.2:44721)
0923 14:38:41.916648 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:41.916661 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:41.916747 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:41.916987 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:41.916989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:41.917004 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:41.917113 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.917118 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.917766 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.917770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.918506 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.918514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.919076 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.919079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.919175 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.919191 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 14:38:41.919272 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 14:38:41.919335 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":1}
I20260923 14:38:41.919451 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.916540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:44721 (local address 127.2.218.3:36747)
0923 14:38:41.916650 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.916655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.916735 (+ 80us) server_negotiation.cc:407] Connection header received
0923 14:38:41.916855 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.916859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.916900 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.916971 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:41.917212 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.917661 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.918621 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.918964 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.919071 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.919125 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.919288 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.919359 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:38:41.919689 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.916476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:43471 (local address 127.2.218.2:56671)
0923 14:38:41.916800 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:41.916818 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:41.916927 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:41.917191 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:41.917193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:41.917206 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:41.917331 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.917336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.917967 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.917972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.918704 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:41.918712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.919328 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.919331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:41.919410 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.919428 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 14:38:41.919492 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 14:38:41.919570 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":28,"thread_start_us":96,"threads_started":1}
I20260923 14:38:41.919970 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.916540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:56671 (local address 127.2.218.1:43471)
0923 14:38:41.916772 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.916776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.916895 (+ 119us) server_negotiation.cc:407] Connection header received
0923 14:38:41.917031 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.917040 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.917088 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.917184 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:41.917440 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.917868 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.918831 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.919203 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.919309 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.919450 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.919751 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.919853 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":62,"threads_started":1}
I20260923 14:38:41.920020 17643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514539aadb53489f8cde29b4646a86c7" candidate_uuid: "9235ee37ce7d4d1788898e1bed052ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" is_pre_election: true
I20260923 14:38:41.920280 17643 raft_consensus.cc:2454] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9235ee37ce7d4d1788898e1bed052ca8 in term 0.
I20260923 14:38:41.920341 17373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514539aadb53489f8cde29b4646a86c7" candidate_uuid: "9235ee37ce7d4d1788898e1bed052ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995089815202416fb2af2712aae7ef4b" is_pre_election: true
I20260923 14:38:41.920600 17373 raft_consensus.cc:2454] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9235ee37ce7d4d1788898e1bed052ca8 in term 0.
I20260923 14:38:41.920732 17447 leader_election.cc:302] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e42dfe32eb64f61ac30a83b88bd8ce7, 9235ee37ce7d4d1788898e1bed052ca8; no voters:
I20260923 14:38:41.920979 17704 raft_consensus.cc:2781] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:38:41.921062 17704 raft_consensus.cc:482] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:38:41.921090 17704 raft_consensus.cc:3037] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:41.922019 17704 raft_consensus.cc:504] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.922359 17704 leader_election.cc:288] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [CANDIDATE]: Term 1 election: Requested vote from peers 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747), 995089815202416fb2af2712aae7ef4b (127.2.218.1:43471)
I20260923 14:38:41.922626 17643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514539aadb53489f8cde29b4646a86c7" candidate_uuid: "9235ee37ce7d4d1788898e1bed052ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7"
I20260923 14:38:41.922644 17373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514539aadb53489f8cde29b4646a86c7" candidate_uuid: "9235ee37ce7d4d1788898e1bed052ca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995089815202416fb2af2712aae7ef4b"
I20260923 14:38:41.922750 17643 raft_consensus.cc:3037] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:41.922760 17373 raft_consensus.cc:3037] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:41.923576 17373 raft_consensus.cc:2454] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9235ee37ce7d4d1788898e1bed052ca8 in term 1.
I20260923 14:38:41.923694 17643 raft_consensus.cc:2454] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9235ee37ce7d4d1788898e1bed052ca8 in term 1.
I20260923 14:38:41.923804 17446 leader_election.cc:302] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9235ee37ce7d4d1788898e1bed052ca8, 995089815202416fb2af2712aae7ef4b; no voters:
I20260923 14:38:41.923940 17704 raft_consensus.cc:2781] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:38:41.924124 17704 raft_consensus.cc:686] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 1 LEADER]: Becoming Leader. State: Replica: 9235ee37ce7d4d1788898e1bed052ca8, State: Running, Role: LEADER
I20260923 14:38:41.924407 17704 consensus_queue.cc:237] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:41.926981 17231 catalog_manager.cc:5495] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2). New cstate: current_term: 1 leader_uuid: "9235ee37ce7d4d1788898e1bed052ca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } health_report { overall_health: UNKNOWN } } }
W20260923 14:38:41.939635 17690 tablet.cc:2369] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:38:41.961615 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:41.958836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.2.218.2:40201)
0923 14:38:41.958994 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:41.958997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:41.959025 (+ 28us) server_negotiation.cc:407] Connection header received
0923 14:38:41.959091 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:41.959092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:41.959136 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:41.959211 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:38:41.959477 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.959875 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.960532 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:41.960702 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:41.960772 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:41.960991 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:38:41.961332 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:38:41.961416 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:41.961476 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 14:38:41.961517 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 14:38:41.966451 17643 raft_consensus.cc:1261] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 1 FOLLOWER]: Refusing update from remote peer 9235ee37ce7d4d1788898e1bed052ca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:38:41.966526 17373 raft_consensus.cc:1261] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Refusing update from remote peer 9235ee37ce7d4d1788898e1bed052ca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:38:41.967077 17704 consensus_queue.cc:1035] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:38:41.967212 17707 consensus_queue.cc:1035] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:38:41.972007 17710 mvcc.cc:205] Tried to move back new op lower bound from 7332554022768386048 to 7332554022604349440. Current Snapshot: MvccSnapshot[applied={T|T < 7332554022768386048}]
I20260923 14:38:41.973389 17711 mvcc.cc:205] Tried to move back new op lower bound from 7332554022768386048 to 7332554022604349440. Current Snapshot: MvccSnapshot[applied={T|T < 7332554022768386048}]
I20260923 14:38:41.974794 17712 mvcc.cc:205] Tried to move back new op lower bound from 7332554022768386048 to 7332554022604349440. Current Snapshot: MvccSnapshot[applied={T|T < 7332554022768386048}]
I20260923 14:38:41.979664 17231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:38:41.979770 17231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:38:41.981680 17231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Sending DeleteTablet for 3 replicas of tablet 514539aadb53489f8cde29b4646a86c7
I20260923 14:38:41.982297 17353 tablet_service.cc:1514] Processing DeleteTablet for tablet 514539aadb53489f8cde29b4646a86c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:41 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:33934
I20260923 14:38:41.982347 17623 tablet_service.cc:1514] Processing DeleteTablet for tablet 514539aadb53489f8cde29b4646a86c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:41 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:48532
I20260923 14:38:41.982389 17491 tablet_service.cc:1514] Processing DeleteTablet for tablet 514539aadb53489f8cde29b4646a86c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:41 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:59096
I20260923 14:38:41.982689 17720 tablet_replica.cc:330] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: stopping tablet replica
I20260923 14:38:41.982837 17720 raft_consensus.cc:2229] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:38:41.982851 17722 tablet_replica.cc:330] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: stopping tablet replica
I20260923 14:38:41.982930 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 17269
I20260923 14:38:41.982999 17720 raft_consensus.cc:2258] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:38:41.984557 17722 raft_consensus.cc:2229] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:38:41.984843 17722 raft_consensus.cc:2258] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:38:41.985414 17721 tablet_replica.cc:330] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: stopping tablet replica
I20260923 14:38:41.985636 17721 raft_consensus.cc:2229] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:38:41.985870 17721 raft_consensus.cc:2258] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:38:41.986215 17722 ts_tablet_manager.cc:1905] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:38:41.987136 17721 ts_tablet_manager.cc:1905] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:38:41.988715 17722 ts_tablet_manager.cc:1918] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:38:41.988859 17722 log.cc:1196] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/514539aadb53489f8cde29b4646a86c7
I20260923 14:38:41.989171 17722 ts_tablet_manager.cc:1939] T 514539aadb53489f8cde29b4646a86c7 P 9235ee37ce7d4d1788898e1bed052ca8: Deleting consensus metadata
I20260923 14:38:41.989622 17721 ts_tablet_manager.cc:1918] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:38:41.989718 17721 log.cc:1196] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/514539aadb53489f8cde29b4646a86c7
I20260923 14:38:41.990029 17721 ts_tablet_manager.cc:1939] T 514539aadb53489f8cde29b4646a86c7 P 7e42dfe32eb64f61ac30a83b88bd8ce7: Deleting consensus metadata
W20260923 14:38:41.990293 17217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:43471 (error 108)
I20260923 14:38:41.990773 17218 catalog_manager.cc:4841] TS 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201): tablet 514539aadb53489f8cde29b4646a86c7 (table test-table [id=960bf78502b94fccabd5feb2931e3a4e]) successfully deleted
I20260923 14:38:41.991120 17218 catalog_manager.cc:4841] TS 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747): tablet 514539aadb53489f8cde29b4646a86c7 (table test-table [id=960bf78502b94fccabd5feb2931e3a4e]) successfully deleted
W20260923 14:38:41.991451 17696 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:38:41.991047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:43471 (local address 127.0.0.1:33938)
0923 14:38:41.991299 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:41.991365 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:43471: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":188}
I20260923 14:38:41.991497 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 17424
W20260923 14:38:41.991590 17217 catalog_manager.cc:4584] TS 995089815202416fb2af2712aae7ef4b (127.2.218.1:43471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 514539aadb53489f8cde29b4646a86c7: Network error: Client connection negotiation failed: client connection to 127.2.218.1:43471: connect: Connection reset by peer (error 104)
I20260923 14:38:41.996057 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 17560
I20260923 14:38:42.000653 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 17198
I20260923 14:38:42.005924 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:45857
--webserver_interface=127.2.218.62
--webserver_port=38531
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:45857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:42.105736 17724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:42.105934 17724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:42.105968 17724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:42.109220 17724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:38:42.109268 17724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:42.109288 17724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:42.109306 17724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:38:42.109323 17724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:38:42.113796 17724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:45857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:45857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=38531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:42.114534 17724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:42.115280 17724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:42.122407 17729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:42.122385 17732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:42.122519 17724 server_base.cc:1034] running on GCE node
W20260923 14:38:42.122387 17730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:42.123200 17724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:42.124027 17724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:42.125168 17724 hybrid_clock.cc:648] HybridClock initialized: now 1790174322125167 us; error 33 us; skew 500 ppm
Sep 23 14:38:42 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174322, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:42.127806 17724 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:38:42.128502 17724 webserver.cc:460] Webserver started at http://127.2.218.62:38531/ using document root <none> and password file <none>
I20260923 14:38:42.128924 17724 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:42.128983 17724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:42.132337 17724 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:38:42.134263 17739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:42.135315 17724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:42.135444 17724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "96e737264c3b40e3bcc193b5eeacbb4a"
format_stamp: "Formatted at 2026-09-23 14:38:39 on dist-test-slave-zl47"
server_key: "ccbc395566012442c2019b372b9874db"
server_key_iv: "6d4ba78025a56d3309fcfbc9868ac80f"
server_key_version: "encryptionkey@0"
I20260923 14:38:42.135869 17724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:42.147876 17724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:42.150190 17724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:42.150374 17724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:42.156464 17724 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:45857
I20260923 14:38:42.156678 17791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:45857 every 8 connection(s)
I20260923 14:38:42.157368 17724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:38:42.160642 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 17724
I20260923 14:38:42.160840 17792 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:38:42.161096 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:43471
--local_ip_for_outbound_sockets=127.2.218.1
--tserver_master_addrs=127.2.218.62:45857
--webserver_port=45935
--webserver_interface=127.2.218.1
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:38:42.162408 17792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Bootstrap starting.
I20260923 14:38:42.178485 17792 log.cc:824] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:42.188396 17792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:38:42.188742 17792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a: Bootstrap complete.
I20260923 14:38:42.192202 17792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:42.192528 17792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96e737264c3b40e3bcc193b5eeacbb4a, State: Initialized, Role: FOLLOWER
I20260923 14:38:42.192937 17792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:42.193035 17792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:38:42.193110 17792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:38:42.193207 17792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 2 FOLLOWER]: Advancing to term 3
I20260923 14:38:42.194799 17792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:42.195082 17792 leader_election.cc:302] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96e737264c3b40e3bcc193b5eeacbb4a; no voters:
I20260923 14:38:42.195389 17792 leader_election.cc:288] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 14:38:42.195451 17796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 3 FOLLOWER]: Leader election won for term 3
I20260923 14:38:42.195613 17796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [term 3 LEADER]: Becoming Leader. State: Replica: 96e737264c3b40e3bcc193b5eeacbb4a, State: Running, Role: LEADER
I20260923 14:38:42.195919 17796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } }
I20260923 14:38:42.196501 17792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:38:42.196729 17796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96e737264c3b40e3bcc193b5eeacbb4a. Latest consensus state: current_term: 3 leader_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } } }
I20260923 14:38:42.196920 17796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:42.196717 17797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96e737264c3b40e3bcc193b5eeacbb4a" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 45857 } } }
I20260923 14:38:42.197247 17797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96e737264c3b40e3bcc193b5eeacbb4a [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:42.198215 17800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:38:42.202520 17800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:38:42.203213 17800 catalog_manager.cc:1259] Loaded cluster ID: d45d59d8aee54e5e9aaf34d0ed3e002d
I20260923 14:38:42.203266 17800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 14:38:42.204254 17800 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4aa40689ad google::LogMessage::Fail() at ??:0
@ 0x7f4aa406b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4aa40684de google::LogMessage::Flush() at ??:0
@ 0x7f4aa4069a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4aae9a7fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4aae9a8d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4aae9a5579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4aae9e9640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4aa62ba3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4aa62bac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4aa62bc3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4aa62ab512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4aa78ef6db start_thread at ??:0
@ 0x7f4aa2b6661f clone at ??:0
W20260923 14:38:42.279489 17794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:42.279753 17794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:42.279830 17794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:42.285036 17794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:42.285095 17794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:42.285166 17794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:38:42.291570 17794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:43471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=45935
--tserver_master_addrs=127.2.218.62:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:42.292485 17794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:42.293514 17794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:42.302495 17834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:42.302549 17836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:42.302475 17833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:42.303102 17794 server_base.cc:1034] running on GCE node
I20260923 14:38:42.303453 17794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:42.304070 17794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:42.305258 17794 hybrid_clock.cc:648] HybridClock initialized: now 1790174322305235 us; error 29 us; skew 500 ppm
Sep 23 14:38:42 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174322, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:42.308347 17794 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:38:42.309010 17794 webserver.cc:460] Webserver started at http://127.2.218.1:45935/ using document root <none> and password file <none>
I20260923 14:38:42.309458 17794 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:42.309509 17794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:42.312736 17794 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:42.314582 17843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:42.315608 17794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:42.315729 17794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "995089815202416fb2af2712aae7ef4b"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "4789cb2bfded6d3d5d30fdbd5a6fe63a"
server_key_iv: "2a196b8eb4e31c7a65713df5ee403aa4"
server_key_version: "encryptionkey@0"
I20260923 14:38:42.316134 17794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:42.339506 17794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:42.342183 17794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:42.342345 17794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:42.342813 17794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:42.343690 17850 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 14:38:42.348068 17794 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 14:38:42.348130 17794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260923 14:38:42.348210 17794 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 14:38:42.349547 17794 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 14:38:42.349609 17794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 14:38:42.349870 17850 tablet_bootstrap.cc:492] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Bootstrap starting.
I20260923 14:38:42.359184 17794 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:43471
I20260923 14:38:42.359411 17957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:43471 every 8 connection(s)
I20260923 14:38:42.360087 17794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:38:42.366992 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 17794
I20260923 14:38:42.367354 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:40201
--local_ip_for_outbound_sockets=127.2.218.2
--tserver_master_addrs=127.2.218.62:45857
--webserver_port=44261
--webserver_interface=127.2.218.2
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:38:42.369200 17850 log.cc:824] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:42.376744 17850 tablet_bootstrap.cc:492] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:38:42.377074 17850 tablet_bootstrap.cc:492] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Bootstrap complete.
I20260923 14:38:42.378288 17850 ts_tablet_manager.cc:1397] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.012s
I20260923 14:38:42.380925 17850 raft_consensus.cc:348] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:42.381407 17850 raft_consensus.cc:729] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995089815202416fb2af2712aae7ef4b, State: Initialized, Role: FOLLOWER
I20260923 14:38:42.381857 17850 consensus_queue.cc:260] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:42.382719 17850 ts_tablet_manager.cc:1428] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
W20260923 14:38:42.402069 17960 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:38:42.361757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:42285)
0923 14:38:42.362190 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:42.362235 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:42.362904 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:42.401335 (+ 38431us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":123,"threads_started":1}
W20260923 14:38:42.402586 17958 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:45857 (0 consecutive failures): Network error: Failed to ping master at 127.2.218.62:45857: Client connection negotiation failed: client connection to 127.2.218.62:45857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 14:38:42.403196 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:42.402994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:57833)
0923 14:38:42.403098 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:42.403151 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 14:38:42.403791 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:42.403609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:41821)
0923 14:38:42.403701 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:42.403740 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 14:38:42.403944 17958 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:38:42.462430 17961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:42.462626 17961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:42.462694 17961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:42.466025 17961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:42.466094 17961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:42.466197 17961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:38:42.470494 17961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:40201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=44261
--tserver_master_addrs=127.2.218.62:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:42.471220 17961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:42.471966 17961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:42.479656 17969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:42.479667 17971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:42.479661 17968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:42.479748 17961 server_base.cc:1034] running on GCE node
I20260923 14:38:42.480192 17961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:42.480681 17961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:42.481910 17961 hybrid_clock.cc:648] HybridClock initialized: now 1790174322481898 us; error 26 us; skew 500 ppm
Sep 23 14:38:42 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174322, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:42.484330 17961 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:38:42.485106 17961 webserver.cc:460] Webserver started at http://127.2.218.2:44261/ using document root <none> and password file <none>
I20260923 14:38:42.485541 17961 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:42.485592 17961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:42.488629 17961 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:42.490274 17978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:42.491307 17961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:42.491402 17961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "9235ee37ce7d4d1788898e1bed052ca8"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "efa63c4afd30dd6ceea517bb143d86e7"
server_key_iv: "3ed740ffd6d3c2b9cc7da4fe2ebed666"
server_key_version: "encryptionkey@0"
I20260923 14:38:42.491852 17961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:42.507633 17961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:42.509681 17961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:42.509836 17961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:42.510324 17961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:42.510942 17961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:42.510980 17961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:42.511035 17961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:42.511067 17961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:42.518335 17961 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:40201
I20260923 14:38:42.518378 18091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:40201 every 8 connection(s)
I20260923 14:38:42.519138 17961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:38:42.521790 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:42.520645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:36935)
0923 14:38:42.521176 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:42.521256 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":86,"threads_started":1}
W20260923 14:38:42.522229 18092 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:45857 (0 consecutive failures): Network error: Failed to ping master at 127.2.218.62:45857: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
I20260923 14:38:42.522657 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:42.522488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:35875)
0923 14:38:42.522575 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:42.522613 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 14:38:42.522835 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 17961
I20260923 14:38:42.523195 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:36747
--local_ip_for_outbound_sockets=127.2.218.3
--tserver_master_addrs=127.2.218.62:45857
--webserver_port=34155
--webserver_interface=127.2.218.3
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:38:42.523295 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:42.523082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:47005)
0923 14:38:42.523185 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:42.523224 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 14:38:42.523506 18092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:38:42.614698 18095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:42.614881 18095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:42.614917 18095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:42.618212 18095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:42.618276 18095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:42.618342 18095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:38:42.622903 18095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:36747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=34155
--tserver_master_addrs=127.2.218.62:45857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:42.623634 18095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:42.624379 18095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:42.632104 18100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:42.632123 18101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:42.632124 18103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:42.632162 18095 server_base.cc:1034] running on GCE node
I20260923 14:38:42.632655 18095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:42.633104 18095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:42.634255 18095 hybrid_clock.cc:648] HybridClock initialized: now 1790174322634229 us; error 37 us; skew 500 ppm
Sep 23 14:38:42 dist-test-slave-zl47 krb5kdc[16651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174322, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:42.636495 18095 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:38:42.637233 18095 webserver.cc:460] Webserver started at http://127.2.218.3:34155/ using document root <none> and password file <none>
I20260923 14:38:42.637640 18095 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:42.637696 18095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:42.640780 18095 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:42.642441 18110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:42.643416 18095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:42.643510 18095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7"
format_stamp: "Formatted at 2026-09-23 14:38:40 on dist-test-slave-zl47"
server_key: "9270ced40adf72678564be703e17c137"
server_key_iv: "9e2d28af5caeec94c35310a44219153c"
server_key_version: "encryptionkey@0"
I20260923 14:38:42.643913 18095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:42.655416 18095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:42.657570 18095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:42.657691 18095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:42.658089 18095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:42.658784 18095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:42.658856 18095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:42.658888 18095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:42.658913 18095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:42.666965 18095 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:36747
I20260923 14:38:42.666985 18223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:36747 every 8 connection(s)
I20260923 14:38:42.667837 18095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:38:42.668267 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 18095
I20260923 14:38:42.671020 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:42.669570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:35569)
0923 14:38:42.670381 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:42.670466 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":497,"threads_started":1}
W20260923 14:38:42.671489 18224 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:45857 (0 consecutive failures): Network error: Failed to ping master at 127.2.218.62:45857: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
I20260923 14:38:42.672026 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:42.671846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:58073)
0923 14:38:42.671937 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:42.671979 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 14:38:42.672617 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:42.672438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:40317)
0923 14:38:42.672530 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:42.672575 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 14:38:42.672782 18224 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 14:38:43.404690 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:43.404377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:32991)
0923 14:38:43.404567 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:43.404614 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":57,"threads_started":1}
I20260923 14:38:43.524343 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:43.523979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:41551)
0923 14:38:43.524190 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:43.524238 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260923 14:38:43.673530 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:43.673222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:59301)
0923 14:38:43.673411 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:43.673451 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
I20260923 14:38:44.206794 18231 raft_consensus.cc:482] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:38:44.207012 18231 raft_consensus.cc:504] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:44.207998 18231 leader_election.cc:288] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747), 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201)
Sep 23 14:38:44 dist-test-slave-zl47 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174322, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.2@KRBTEST.COM
Sep 23 14:38:44 dist-test-slave-zl47 krb5kdc[16651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174322, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.3@KRBTEST.COM
I20260923 14:38:44.221624 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:44.208338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:58207 (local address 127.2.218.3:36747)
0923 14:38:44.208715 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:44.208732 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:44.208776 (+ 44us) server_negotiation.cc:407] Connection header received
0923 14:38:44.209505 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:44.209511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:44.209874 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:44.210158 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:44.211417 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:44.212294 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:44.213453 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:44.213834 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:44.213994 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:44.216785 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:44.216809 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:44.216823 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:44.216855 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:44.218776 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:44.219313 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:44.219318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:44.219323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:44.219455 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:44.219857 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:44.219860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:44.219861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:44.220293 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:44.220445 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:44.221182 (+ 737us) server_negotiation.cc:299] Negotiation successful
0923 14:38:44.221450 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":102,"threads_started":1}
I20260923 14:38:44.221625 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:44.208800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:39471 (local address 127.2.218.2:40201)
0923 14:38:44.209151 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:44.209181 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:44.209219 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:38:44.209829 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:44.209834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:44.210202 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:44.210507 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:44.211394 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:44.212304 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:44.213419 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:44.213846 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:44.214009 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:44.216424 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:44.216429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:44.216439 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:44.216461 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:44.218619 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:44.219313 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:44.219318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:44.219323 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:44.219455 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:44.219862 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:44.219865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:44.219867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:44.220293 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:44.220445 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:44.221182 (+ 737us) server_negotiation.cc:299] Negotiation successful
0923 14:38:44.221450 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":108,"threads_started":1}
I20260923 14:38:44.221619 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:44.208229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:36747 (local address 127.2.218.1:58207)
0923 14:38:44.208489 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:44.208504 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:44.208603 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:44.210246 (+ 1643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:44.210252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:44.210769 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:44.211223 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:44.211239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:44.212450 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:44.212455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:44.213321 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:44.213331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:44.213956 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:44.213960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:44.214176 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:44.214773 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:44.214789 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:44.216599 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:44.218870 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:44.218872 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:44.218874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:44.219201 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:44.219551 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:44.219554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:44.219555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:44.219752 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:44.220437 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:44.220443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:44.220796 (+ 353us) client_negotiation.cc:769] Sending connection context
0923 14:38:44.221010 (+ 214us) client_negotiation.cc:240] Negotiation successful
0923 14:38:44.221407 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":151,"spinlock_wait_cycles":20864,"thread_start_us":117,"threads_started":1}
I20260923 14:38:44.221618 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:44.208632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40201 (local address 127.2.218.1:39471)
0923 14:38:44.208920 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:44.208956 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:44.209072 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:44.210472 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:44.210475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:44.210766 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:44.211224 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:44.211238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:44.212453 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:44.212455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:44.213302 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:44.213311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:44.213978 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:44.213981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:44.214176 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:44.214814 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:44.214826 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:44.216226 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:44.218729 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:44.218733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:44.218742 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:44.219201 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:44.219551 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:44.219554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:44.219555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:44.219752 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:44.220437 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:44.220443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:44.220785 (+ 342us) client_negotiation.cc:769] Sending connection context
0923 14:38:44.221010 (+ 225us) client_negotiation.cc:240] Negotiation successful
0923 14:38:44.221462 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":360,"spinlock_wait_cycles":36096,"thread_start_us":73,"threads_started":1}
W20260923 14:38:44.224116 17847 leader_election.cc:341] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747): Not found: Tablet not found: 514539aadb53489f8cde29b4646a86c7
W20260923 14:38:44.224349 17847 leader_election.cc:341] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201): Not found: Tablet not found: 514539aadb53489f8cde29b4646a86c7
I20260923 14:38:44.224426 17847 leader_election.cc:302] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 995089815202416fb2af2712aae7ef4b; no voters: 7e42dfe32eb64f61ac30a83b88bd8ce7, 9235ee37ce7d4d1788898e1bed052ca8
I20260923 14:38:44.224622 18231 raft_consensus.cc:2726] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:38:44.405553 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:44.405314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:43617)
0923 14:38:44.405430 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:44.405485 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 14:38:44.525301 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:44.524952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:50321)
0923 14:38:44.525163 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:44.525210 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260923 14:38:44.674486 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:44.674152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:56819)
0923 14:38:44.674348 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:44.674397 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":55,"threads_started":1}
I20260923 14:38:45.406512 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:45.406169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:46609)
0923 14:38:45.406384 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:45.406432 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":60,"threads_started":1}
I20260923 14:38:45.526257 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:45.525947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:57955)
0923 14:38:45.526149 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:45.526190 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":62,"threads_started":1}
I20260923 14:38:45.675359 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:45.675043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:37073)
0923 14:38:45.675239 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:45.675283 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":56,"threads_started":1}
I20260923 14:38:45.990027 18241 raft_consensus.cc:482] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:38:45.990178 18241 raft_consensus.cc:504] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:45.990588 18241 leader_election.cc:288] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747), 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201)
W20260923 14:38:45.991346 17847 leader_election.cc:341] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201): Not found: Tablet not found: 514539aadb53489f8cde29b4646a86c7
W20260923 14:38:45.991467 17847 leader_election.cc:341] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747): Not found: Tablet not found: 514539aadb53489f8cde29b4646a86c7
I20260923 14:38:45.991539 17847 leader_election.cc:302] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 995089815202416fb2af2712aae7ef4b; no voters: 7e42dfe32eb64f61ac30a83b88bd8ce7, 9235ee37ce7d4d1788898e1bed052ca8
I20260923 14:38:45.991685 18241 raft_consensus.cc:2726] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:38:46.407397 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:46.407074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:59625)
0923 14:38:46.407266 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:46.407309 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260923 14:38:46.527159 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:46.526850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:55969)
0923 14:38:46.527052 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:46.527095 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":56,"threads_started":1}
I20260923 14:38:46.676416 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:46.676085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:37993)
0923 14:38:46.676297 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:46.676344 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260923 14:38:47.408421 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:47.408055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:53831)
0923 14:38:47.408281 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:47.408328 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":62,"threads_started":1}
I20260923 14:38:47.528020 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:47.527701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:32973)
0923 14:38:47.527905 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:47.527950 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":59,"threads_started":1}
I20260923 14:38:47.677376 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:47.677043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:42769)
0923 14:38:47.677249 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:47.677294 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
I20260923 14:38:48.409458 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:48.409077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:37457)
0923 14:38:48.409322 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:48.409371 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260923 14:38:48.529006 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:48.528644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:40837)
0923 14:38:48.528874 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:48.528919 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260923 14:38:48.678457 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:48.678050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:35559)
0923 14:38:48.678326 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:48.678376 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
I20260923 14:38:49.357033 18251 raft_consensus.cc:482] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:38:49.357152 18251 raft_consensus.cc:504] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:49.357537 18251 leader_election.cc:288] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747), 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201)
W20260923 14:38:49.358343 17847 leader_election.cc:341] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201): Not found: Tablet not found: 514539aadb53489f8cde29b4646a86c7
W20260923 14:38:49.358520 17847 leader_election.cc:341] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747): Not found: Tablet not found: 514539aadb53489f8cde29b4646a86c7
I20260923 14:38:49.358592 17847 leader_election.cc:302] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 995089815202416fb2af2712aae7ef4b; no voters: 7e42dfe32eb64f61ac30a83b88bd8ce7, 9235ee37ce7d4d1788898e1bed052ca8
I20260923 14:38:49.358798 18251 raft_consensus.cc:2726] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:38:49.410509 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:49.410167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:54779)
0923 14:38:49.410362 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:49.410408 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
I20260923 14:38:49.530462 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:49.530039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:34981)
0923 14:38:49.530345 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:49.530390 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260923 14:38:49.679580 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:49.679120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:50931)
0923 14:38:49.679457 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:49.679504 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":78,"threads_started":1}
I20260923 14:38:50.411484 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:50.411110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:34323)
0923 14:38:50.411354 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:50.411399 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":65,"threads_started":1}
I20260923 14:38:50.531388 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:50.531051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:47859)
0923 14:38:50.531269 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:50.531312 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
I20260923 14:38:50.680634 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:50.680257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:38325)
0923 14:38:50.680495 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:50.680549 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260923 14:38:51.275504 18258 raft_consensus.cc:482] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:38:51.275624 18258 raft_consensus.cc:504] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e42dfe32eb64f61ac30a83b88bd8ce7" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36747 } } peers { permanent_uuid: "9235ee37ce7d4d1788898e1bed052ca8" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40201 } } peers { permanent_uuid: "995089815202416fb2af2712aae7ef4b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43471 } }
I20260923 14:38:51.276049 18258 leader_election.cc:288] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747), 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201)
W20260923 14:38:51.276777 17847 leader_election.cc:341] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e42dfe32eb64f61ac30a83b88bd8ce7 (127.2.218.3:36747): Not found: Tablet not found: 514539aadb53489f8cde29b4646a86c7
W20260923 14:38:51.276913 17847 leader_election.cc:341] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9235ee37ce7d4d1788898e1bed052ca8 (127.2.218.2:40201): Not found: Tablet not found: 514539aadb53489f8cde29b4646a86c7
I20260923 14:38:51.276954 17847 leader_election.cc:302] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 995089815202416fb2af2712aae7ef4b; no voters: 7e42dfe32eb64f61ac30a83b88bd8ce7, 9235ee37ce7d4d1788898e1bed052ca8
I20260923 14:38:51.277076 18258 raft_consensus.cc:2726] T 514539aadb53489f8cde29b4646a86c7 P 995089815202416fb2af2712aae7ef4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 14:38:51.412458 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:51.412109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:60843)
0923 14:38:51.412332 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:51.412377 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
I20260923 14:38:51.532366 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:51.531992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:57015)
0923 14:38:51.532224 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:51.532273 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
I20260923 14:38:51.681680 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:51.681301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:34911)
0923 14:38:51.681548 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:51.681599 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
I20260923 14:38:52.413563 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:52.413194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:44191)
0923 14:38:52.413419 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:52.413472 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20260923 14:38:52.533458 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:52.533079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:59125)
0923 14:38:52.533331 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:52.533378 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
I20260923 14:38:52.682754 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:52.682391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:55825)
0923 14:38:52.682603 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:52.682654 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":60,"threads_started":1}
I20260923 14:38:53.414661 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:53.414326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:38101)
0923 14:38:53.414538 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:53.414581 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
I20260923 14:38:53.534435 18266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:53.534111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:34411)
0923 14:38:53.534313 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:53.534359 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
I20260923 14:38:53.683743 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:53.683412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:42723)
0923 14:38:53.683623 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:53.683667 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
I20260923 14:38:54.415644 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:54.415264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:37039)
0923 14:38:54.415495 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:54.415550 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260923 14:38:54.535501 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:54.535095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:40513)
0923 14:38:54.535358 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:54.535407 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260923 14:38:54.684783 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:54.684367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:37495)
0923 14:38:54.684642 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:54.684704 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260923 14:38:55.416642 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:55.416279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:48637)
0923 14:38:55.416518 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:55.416562 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260923 14:38:55.536515 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:55.536176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:49453)
0923 14:38:55.536395 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:55.536441 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":59,"threads_started":1}
I20260923 14:38:55.685777 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:55.685437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:57821)
0923 14:38:55.685656 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:55.685703 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":65,"threads_started":1}
I20260923 14:38:56.417585 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:56.417266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:53147)
0923 14:38:56.417468 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:56.417512 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260923 14:38:56.537468 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:56.537110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:45429)
0923 14:38:56.537340 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:56.537389 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
I20260923 14:38:56.686887 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:56.686460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.3:54531)
0923 14:38:56.686711 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:56.686776 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260923 14:38:57.418634 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:57.418310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.1:43583)
0923 14:38:57.418507 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:57.418553 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260923 14:38:57.538511 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:57.538183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:45857 (local address 127.2.218.2:43461)
0923 14:38:57.538395 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:57.538439 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:45857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260923 14:38:57.671555 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 17794
I20260923 14:38:57.676803 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 17961
I20260923 14:38:57.681542 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 18095
2026-09-23T14:38:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18956 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:38:57 dist-test-slave-zl47 krb5kdc[18281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:38:57 dist-test-slave-zl47 krb5kdc[18281](info): set up 2 sockets
Sep 23 14:38:57 dist-test-slave-zl47 krb5kdc[18281](info): commencing operation
krb5kdc: starting...
W20260923 14:38:58.582489 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.880s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:38:58 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:38:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:38:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:38:58.681157 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33485
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:33485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:58.775075 18294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:58.775265 18294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:58.775329 18294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:58.778890 18294 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 14:38:58.778931 18294 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 14:38:58.778949 18294 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 14:38:58.779002 18294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:38:58.779037 18294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:58.779079 18294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:58.779142 18294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:38:58.779181 18294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:38:58.783428 18294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:33485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:58.784181 18294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:58.785074 18294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:58.792218 18299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:58.792255 18294 server_base.cc:1034] running on GCE node
W20260923 14:38:58.792227 18302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:58.792229 18300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:58.792994 18294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:58.793965 18294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:58.795141 18294 hybrid_clock.cc:648] HybridClock initialized: now 1790174338795120 us; error 43 us; skew 500 ppm
Sep 23 14:38:58 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174338, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:58.797447 18294 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:38:58.798213 18294 webserver.cc:460] Webserver started at http://127.2.218.62:39597/ using document root <none> and password file <none>
I20260923 14:38:58.798650 18294 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:58.798729 18294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:58.798908 18294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:58.800496 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "96a52d6c518b4af0a1e5f194c2760b0b"
format_stamp: "Formatted at 2026-09-23 14:38:58 on dist-test-slave-zl47"
server_key: "9a890ecf853ba7d8fcc056c3301c5b75"
server_key_iv: "ddd578590bb847894cae7bc2fd817b37"
server_key_version: "encryptionkey@0"
I20260923 14:38:58.801064 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "96a52d6c518b4af0a1e5f194c2760b0b"
format_stamp: "Formatted at 2026-09-23 14:38:58 on dist-test-slave-zl47"
server_key: "9a890ecf853ba7d8fcc056c3301c5b75"
server_key_iv: "ddd578590bb847894cae7bc2fd817b37"
server_key_version: "encryptionkey@0"
I20260923 14:38:58.804195 18294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:38:58.806267 18309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:58.807195 18294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:38:58.807315 18294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "96a52d6c518b4af0a1e5f194c2760b0b"
format_stamp: "Formatted at 2026-09-23 14:38:58 on dist-test-slave-zl47"
server_key: "9a890ecf853ba7d8fcc056c3301c5b75"
server_key_iv: "ddd578590bb847894cae7bc2fd817b37"
server_key_version: "encryptionkey@0"
I20260923 14:38:58.807408 18294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:58.821535 18294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:58.824028 18294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:58.824190 18294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:58.829818 18294 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:33485
I20260923 14:38:58.830065 18361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:33485 every 8 connection(s)
I20260923 14:38:58.830566 18294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:38:58.832706 18362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:58.835573 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 18294
I20260923 14:38:58.835686 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:38:58.835922 2920 external_mini_cluster.cc:1428] Setting key b0a324e5af118df2d6ea7ce91a36715f
I20260923 14:38:58.837358 18362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Bootstrap starting.
I20260923 14:38:58.839372 18362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:58.839986 18362 log.cc:824] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:58.841639 18362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: No bootstrap required, opened a new log
Sep 23 14:38:58 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:58.844910 18362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:38:58.845096 18362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:58.845166 18362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a52d6c518b4af0a1e5f194c2760b0b, State: Initialized, Role: FOLLOWER
I20260923 14:38:58.845505 18362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:38:58.845631 18362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:38:58.845687 18362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:38:58.845767 18362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:58.846508 18362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:38:58.846782 18362 leader_election.cc:302] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a52d6c518b4af0a1e5f194c2760b0b; no voters:
I20260923 14:38:58.847056 18362 leader_election.cc:288] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:38:58.847123 18367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:38:58.847463 18367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 1 LEADER]: Becoming Leader. State: Replica: 96a52d6c518b4af0a1e5f194c2760b0b, State: Running, Role: LEADER
I20260923 14:38:58.847756 18367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:38:58.848160 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:58.837023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.2.218.62:33485)
0923 14:38:58.837348 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:58.837366 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:58.837393 (+ 27us) server_negotiation.cc:407] Connection header received
0923 14:38:58.837937 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:58.837950 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:58.838260 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:58.838498 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:58.839239 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:58.840107 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:58.841070 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:58.841489 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:58.841605 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:58.843932 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:58.843936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:58.843945 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:58.843966 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:58.845541 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:58.845972 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:58.845976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:58.845981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:58.846081 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:58.846370 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:58.846373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:58.846375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:58.846716 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:58.846898 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:58.847115 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 14:38:58.847385 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
I20260923 14:38:58.848178 18362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:38:58.848807 18367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a52d6c518b4af0a1e5f194c2760b0b. Latest consensus state: current_term: 1 leader_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } } }
I20260923 14:38:58.848925 18367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:58.849227 18368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } } }
I20260923 14:38:58.849488 18368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: This master's current role is: LEADER
I20260923 14:38:58.850234 18375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:38:58.852993 18375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:38:58.857882 18375 catalog_manager.cc:1347] Generated new cluster ID: 97ce71c1ff5f42bab62d07db9a080615
I20260923 14:38:58.857936 18375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:38:58.865869 18375 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:38:58.866604 18375 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:38:58.878775 18375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Generated new TSK 0
I20260923 14:38:58.879228 18375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:38:58.908128 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33485
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:59.016904 18388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:59.017093 18388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:59.017127 18388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:59.020560 18388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:59.020604 18388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:59.020689 18388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:38:59.024883 18388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:59.025635 18388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:59.026456 18388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:59.034369 18393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:59.034379 18396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:59.034528 18394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:59.034626 18388 server_base.cc:1034] running on GCE node
I20260923 14:38:59.035046 18388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:59.035569 18388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:59.036852 18388 hybrid_clock.cc:648] HybridClock initialized: now 1790174339036815 us; error 33 us; skew 500 ppm
Sep 23 14:38:59 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174339, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:59.039417 18388 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:38:59.040098 18388 webserver.cc:460] Webserver started at http://127.2.218.1:38295/ using document root <none> and password file <none>
I20260923 14:38:59.040560 18388 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:59.040642 18388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:59.040844 18388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:59.042524 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "185023dcb9a1476a856c03e7b7150b4e"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "8c2b2c4e409da0b2ce4d1b01019982e5"
server_key_iv: "5f0e9572c12b169f7fb07b4636bbb42b"
server_key_version: "encryptionkey@0"
I20260923 14:38:59.043059 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "185023dcb9a1476a856c03e7b7150b4e"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "8c2b2c4e409da0b2ce4d1b01019982e5"
server_key_iv: "5f0e9572c12b169f7fb07b4636bbb42b"
server_key_version: "encryptionkey@0"
I20260923 14:38:59.046042 18388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:38:59.048040 18403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:59.049070 18388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:38:59.049191 18388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "185023dcb9a1476a856c03e7b7150b4e"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "8c2b2c4e409da0b2ce4d1b01019982e5"
server_key_iv: "5f0e9572c12b169f7fb07b4636bbb42b"
server_key_version: "encryptionkey@0"
I20260923 14:38:59.049288 18388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:59.060560 18388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:59.062784 18388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:59.062948 18388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:59.063391 18388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:59.064024 18388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:59.064090 18388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:59.064146 18388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:59.064177 18388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:59.071395 18388 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:38711
I20260923 14:38:59.071426 18516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:38711 every 8 connection(s)
I20260923 14:38:59.072090 18388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:38:59.073177 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 18388
I20260923 14:38:59.073294 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:38:59.073549 2920 external_mini_cluster.cc:1428] Setting key a60106646ab78a98e467312b2bb3a8cf
Sep 23 14:38:59 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174339, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
I20260923 14:38:59.084784 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.073601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:48125 (local address 127.2.218.62:33485)
0923 14:38:59.073946 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:59.073950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:59.074648 (+ 698us) server_negotiation.cc:407] Connection header received
0923 14:38:59.075494 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:59.075499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:59.075555 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:59.075634 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:59.076819 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.077276 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.078855 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.079060 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.079132 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.081473 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:59.081476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:59.081478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:59.081498 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:59.082805 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:59.083336 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:59.083341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:59.083342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:59.083412 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:59.083709 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:59.083713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:59.083714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:59.083931 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:59.084030 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:59.084507 (+ 477us) server_negotiation.cc:299] Negotiation successful
0923 14:38:59.084626 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260923 14:38:59.085373 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.073919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:48125)
0923 14:38:59.074529 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:59.074570 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:59.075292 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:59.075789 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:59.075803 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:59.076126 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:59.076678 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.076695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.077416 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.077422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.078747 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.078757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.079202 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.079209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.079420 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.079920 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:59.079944 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:59.081325 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:59.082916 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:59.082921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:59.082932 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:59.083236 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:59.083496 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:59.083499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:59.083500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:59.083622 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:59.084030 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:59.084036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:59.084288 (+ 252us) client_negotiation.cc:769] Sending connection context
0923 14:38:59.084462 (+ 174us) client_negotiation.cc:240] Negotiation successful
0923 14:38:59.084718 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":132,"threads_started":1}
I20260923 14:38:59.086428 18517 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33485
I20260923 14:38:59.086629 18517 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:59.086995 18517 heartbeater.cc:511] Master 127.2.218.62:33485 requested a full tablet report, sending...
I20260923 14:38:59.088425 18326 ts_manager.cc:194] Registered new tserver with Master: 185023dcb9a1476a856c03e7b7150b4e (127.2.218.1:38711)
I20260923 14:38:59.089977 18326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:48125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:38:59.098938 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33485
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:59.190357 18522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:59.190568 18522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:59.190629 18522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:59.193955 18522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:59.194016 18522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:59.194103 18522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:38:59.198257 18522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:59.199012 18522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:59.199757 18522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:59.207399 18530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:59.207422 18527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:59.207607 18528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:59.207813 18522 server_base.cc:1034] running on GCE node
I20260923 14:38:59.208139 18522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:59.208638 18522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:59.209930 18522 hybrid_clock.cc:648] HybridClock initialized: now 1790174339209888 us; error 47 us; skew 500 ppm
Sep 23 14:38:59 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174339, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:59.212332 18522 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:38:59.212954 18522 webserver.cc:460] Webserver started at http://127.2.218.2:33079/ using document root <none> and password file <none>
I20260923 14:38:59.213352 18522 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:59.213400 18522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:59.213526 18522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:59.215137 18522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "391cf02b500543258d46f14cec926d29"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "953e60b06c1883677999e787ca4e45c2"
server_key_iv: "07dcaaacd38e84f175d722367edf4f2a"
server_key_version: "encryptionkey@0"
I20260923 14:38:59.215549 18522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "391cf02b500543258d46f14cec926d29"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "953e60b06c1883677999e787ca4e45c2"
server_key_iv: "07dcaaacd38e84f175d722367edf4f2a"
server_key_version: "encryptionkey@0"
I20260923 14:38:59.218423 18522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260923 14:38:59.220223 18537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:59.221225 18522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:38:59.221304 18522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "391cf02b500543258d46f14cec926d29"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "953e60b06c1883677999e787ca4e45c2"
server_key_iv: "07dcaaacd38e84f175d722367edf4f2a"
server_key_version: "encryptionkey@0"
I20260923 14:38:59.221412 18522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:59.240175 18522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:59.242494 18522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:59.242651 18522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:59.243106 18522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:59.243747 18522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:59.243808 18522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:59.243889 18522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:59.243933 18522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:59.251659 18522 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:43593
I20260923 14:38:59.251695 18650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:43593 every 8 connection(s)
I20260923 14:38:59.252391 18522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:38:59.253458 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 18522
I20260923 14:38:59.253573 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:38:59.253857 2920 external_mini_cluster.cc:1428] Setting key bf144a9a4632a94d53b3cdade0646fe8
Sep 23 14:38:59 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174339, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
I20260923 14:38:59.264636 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.253951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:48363 (local address 127.2.218.62:33485)
0923 14:38:59.254078 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:59.254084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:59.255012 (+ 928us) server_negotiation.cc:407] Connection header received
0923 14:38:59.255921 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:59.255923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:59.255961 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:59.256018 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:59.257334 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.257815 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.258942 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.259316 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.259438 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.261470 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:59.261475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:59.261478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:59.261504 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:59.262941 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:59.263350 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:59.263352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:59.263354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:59.263509 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:59.263791 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:59.263793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:59.263793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:59.263944 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:59.264009 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:59.264445 (+ 436us) server_negotiation.cc:299] Negotiation successful
0923 14:38:59.264535 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
I20260923 14:38:59.265494 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.254250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:48363)
0923 14:38:59.254869 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:59.254915 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:59.255715 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:59.256170 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:59.256180 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:59.256644 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:59.257192 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.257207 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.257938 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.257942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.258747 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.258756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.259449 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.259452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.259585 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.260063 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:59.260077 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:59.261322 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:59.263035 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:59.263038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:59.263047 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:59.263263 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:59.263592 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:59.263595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:59.263596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:59.263715 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:59.264022 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:59.264025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:59.264197 (+ 172us) client_negotiation.cc:769] Sending connection context
0923 14:38:59.264372 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 14:38:59.264657 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":123,"threads_started":1}
I20260923 14:38:59.266465 18651 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33485
I20260923 14:38:59.266674 18651 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:59.267123 18651 heartbeater.cc:511] Master 127.2.218.62:33485 requested a full tablet report, sending...
I20260923 14:38:59.267971 18326 ts_manager.cc:194] Registered new tserver with Master: 391cf02b500543258d46f14cec926d29 (127.2.218.2:43593)
I20260923 14:38:59.268700 18326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:48363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:38:59.277887 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33485
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:38:59.376814 18656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:38:59.377027 18656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:38:59.377066 18656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:38:59.380935 18656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:38:59.380980 18656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:38:59.381033 18656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:38:59.385972 18656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:38:59.386830 18656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:38:59.387660 18656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:38:59.396458 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:59.396518 18656 server_base.cc:1034] running on GCE node
W20260923 14:38:59.396539 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:38:59.396454 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:38:59.397115 18656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:38:59.397766 18656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:38:59.399010 18656 hybrid_clock.cc:648] HybridClock initialized: now 1790174339398977 us; error 32 us; skew 500 ppm
Sep 23 14:38:59 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174339, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:38:59.401700 18656 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:38:59.402689 18656 webserver.cc:460] Webserver started at http://127.2.218.3:37239/ using document root <none> and password file <none>
I20260923 14:38:59.403260 18656 fs_manager.cc:359] Metadata directory not provided
I20260923 14:38:59.403366 18656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:38:59.403610 18656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:38:59.405627 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "4437d75272114d0e85dec0cfcf851f3f"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "3a30f4ae3cbb577c8967184b5a7356c4"
server_key_iv: "8d66a243f6bf96191aaa9678fb335ca6"
server_key_version: "encryptionkey@0"
I20260923 14:38:59.406320 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "4437d75272114d0e85dec0cfcf851f3f"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "3a30f4ae3cbb577c8967184b5a7356c4"
server_key_iv: "8d66a243f6bf96191aaa9678fb335ca6"
server_key_version: "encryptionkey@0"
I20260923 14:38:59.410084 18656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:38:59.412606 18671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:59.413810 18656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:38:59.413959 18656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "4437d75272114d0e85dec0cfcf851f3f"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "3a30f4ae3cbb577c8967184b5a7356c4"
server_key_iv: "8d66a243f6bf96191aaa9678fb335ca6"
server_key_version: "encryptionkey@0"
I20260923 14:38:59.414079 18656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:38:59.425829 18656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:38:59.428671 18656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:38:59.428911 18656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:38:59.429536 18656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:38:59.430341 18656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:38:59.430415 18656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:59.430470 18656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:38:59.430521 18656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:38:59.439837 18656 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:32801
I20260923 14:38:59.439862 18784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:32801 every 8 connection(s)
I20260923 14:38:59.440587 18656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:38:59.442586 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 18656
I20260923 14:38:59.442721 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:38:59.443130 2920 external_mini_cluster.cc:1428] Setting key 101ade8416917d56a34d326170597cee
Sep 23 14:38:59 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174339, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:38:59.454896 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.442273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:44797 (local address 127.2.218.62:33485)
0923 14:38:59.442421 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:59.442426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:59.443617 (+ 1191us) server_negotiation.cc:407] Connection header received
0923 14:38:59.444887 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:59.444891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:59.444954 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:59.445049 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:59.446352 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.446958 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.448360 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.448622 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.448789 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.451341 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:59.451345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:59.451347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:59.451373 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:59.452928 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:59.453438 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:59.453441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:59.453443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:59.453514 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:59.453853 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:59.453858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:59.453864 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:59.454026 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:59.454150 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:59.454611 (+ 461us) server_negotiation.cc:299] Negotiation successful
0923 14:38:59.454724 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:38:59.455621 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.442626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:44797)
0923 14:38:59.443438 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:59.443498 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:59.444658 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:59.445218 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:59.445251 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:59.445656 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:38:59.446176 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.446196 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.447104 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.447108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.448189 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.448199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.448789 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.448794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.449019 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.449661 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:38:59.449678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:38:59.451179 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:38:59.453044 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:59.453050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:59.453062 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:59.453312 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:59.453605 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:38:59.453609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:38:59.453611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:38:59.453741 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:38:59.454152 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:38:59.454159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:38:59.454382 (+ 223us) client_negotiation.cc:769] Sending connection context
0923 14:38:59.454552 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 14:38:59.454831 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":130,"threads_started":1}
I20260923 14:38:59.456760 18785 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33485
I20260923 14:38:59.456991 18785 heartbeater.cc:464] Registering TS with master...
I20260923 14:38:59.457443 18785 heartbeater.cc:511] Master 127.2.218.62:33485 requested a full tablet report, sending...
I20260923 14:38:59.458362 18326 ts_manager.cc:194] Registered new tserver with Master: 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801)
I20260923 14:38:59.458890 18326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:44797
I20260923 14:38:59.468631 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:38:59.477097 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.470608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.2.218.62:33485)
0923 14:38:59.470707 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:59.470709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:59.470927 (+ 218us) server_negotiation.cc:407] Connection header received
0923 14:38:59.471024 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:59.471026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:59.471057 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:59.471115 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:38:59.471773 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.472196 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.472962 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.473156 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.473226 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.474488 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:38:59.474490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:38:59.474492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:38:59.474509 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:38:59.475735 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:59.476095 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:59.476097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:59.476098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:59.476161 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:38:59.476434 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:38:59.476436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:38:59.476437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:38:59.476594 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:38:59.476680 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:59.476920 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 14:38:59.476994 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:38:59.479461 18326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:38:59.482200 18326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:38:59.495316 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.489640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:32801 (local address 127.0.0.1:39596)
0923 14:38:59.489986 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:59.490030 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:59.490153 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:59.491176 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:59.491181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:59.491200 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:59.491316 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.491322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.492585 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.492588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.493915 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.493924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.494803 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.494806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.494946 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.494988 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 14:38:59.495089 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 14:38:59.495206 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":130,"thread_start_us":87,"threads_started":1}
I20260923 14:38:59.495330 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.489755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.2.218.3:32801)
0923 14:38:59.490762 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:59.490774 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:59.490797 (+ 23us) server_negotiation.cc:407] Connection header received
0923 14:38:59.490859 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:59.490865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:59.491037 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:59.491185 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:59.491424 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.492460 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.494041 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.494689 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.494800 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.494927 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:59.495115 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 14:38:59.495195 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":115,"threads_started":1}
I20260923 14:38:59.495347 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.490260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:38711 (local address 127.0.0.1:36582)
0923 14:38:59.490550 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:59.490562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:59.490646 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:59.491265 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:59.491267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:59.491282 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:59.491402 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.491408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.492462 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.492467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.493874 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.493883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.494904 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.494907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.495004 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.495032 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 14:38:59.495075 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 14:38:59.495205 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":90,"thread_start_us":103,"threads_started":1}
I20260923 14:38:59.495874 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.490489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.2.218.1:38711)
0923 14:38:59.490864 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:59.490875 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:59.490895 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:38:59.490967 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:59.490971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:59.491125 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:59.491535 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:59.491693 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.492295 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.493990 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.494775 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.495326 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.495626 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:59.495674 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 14:38:59.495750 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":129,"threads_started":1}
I20260923 14:38:59.495888 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.490390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.2.218.2:43593)
0923 14:38:59.490913 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:59.490923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:59.490938 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:38:59.490995 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:59.490998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:59.491125 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:59.491257 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:59.491601 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.492618 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.494127 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.494788 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.494886 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.495003 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:59.495675 (+ 672us) server_negotiation.cc:299] Negotiation successful
0923 14:38:59.495748 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":95,"threads_started":1}
I20260923 14:38:59.496098 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.489902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:43593 (local address 127.0.0.1:41006)
0923 14:38:59.490128 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:59.490139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:59.490205 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:59.491402 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:59.491405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:59.491418 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:59.491498 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.491502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.492971 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.492975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.494013 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.494019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.495054 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.495059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.495132 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.495153 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 14:38:59.495208 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 14:38:59.495258 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260923 14:38:59.497934 18719 tablet_service.cc:1467] Processing CreateTablet for tablet 327ea7091a7043c1a92e789f52c2da6c (DEFAULT_TABLE table=test-table [id=d349bbd7a3d5492d8a20e5f747c33763]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:59.497956 18451 tablet_service.cc:1467] Processing CreateTablet for tablet 327ea7091a7043c1a92e789f52c2da6c (DEFAULT_TABLE table=test-table [id=d349bbd7a3d5492d8a20e5f747c33763]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:59.498216 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 327ea7091a7043c1a92e789f52c2da6c (DEFAULT_TABLE table=test-table [id=d349bbd7a3d5492d8a20e5f747c33763]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:38:59.498908 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327ea7091a7043c1a92e789f52c2da6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:59.498911 18719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327ea7091a7043c1a92e789f52c2da6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:59.499089 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327ea7091a7043c1a92e789f52c2da6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:38:59.504333 18802 tablet_bootstrap.cc:492] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: Bootstrap starting.
I20260923 14:38:59.505477 18803 tablet_bootstrap.cc:492] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Bootstrap starting.
I20260923 14:38:59.506394 18804 tablet_bootstrap.cc:492] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: Bootstrap starting.
I20260923 14:38:59.507198 18802 tablet_bootstrap.cc:654] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:59.507601 18803 tablet_bootstrap.cc:654] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:59.507891 18802 log.cc:824] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:59.508231 18804 tablet_bootstrap.cc:654] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: Neither blocks nor log segments found. Creating new log.
I20260923 14:38:59.508354 18803 log.cc:824] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:59.509269 18804 log.cc:824] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:38:59.509765 18802 tablet_bootstrap.cc:492] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: No bootstrap required, opened a new log
I20260923 14:38:59.509938 18802 ts_tablet_manager.cc:1397] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:38:59.510195 18803 tablet_bootstrap.cc:492] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: No bootstrap required, opened a new log
I20260923 14:38:59.510382 18803 ts_tablet_manager.cc:1397] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:38:59.511042 18804 tablet_bootstrap.cc:492] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: No bootstrap required, opened a new log
I20260923 14:38:59.511204 18804 ts_tablet_manager.cc:1397] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:38:59.512984 18802 raft_consensus.cc:348] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:38:59.513196 18802 raft_consensus.cc:374] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:59.513253 18802 raft_consensus.cc:729] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391cf02b500543258d46f14cec926d29, State: Initialized, Role: FOLLOWER
I20260923 14:38:59.513634 18802 consensus_queue.cc:260] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:38:59.513729 18803 raft_consensus.cc:348] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:38:59.513964 18803 raft_consensus.cc:374] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:59.514024 18803 raft_consensus.cc:729] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185023dcb9a1476a856c03e7b7150b4e, State: Initialized, Role: FOLLOWER
I20260923 14:38:59.514258 18651 heartbeater.cc:503] Master 127.2.218.62:33485 was elected leader, sending a full tablet report...
I20260923 14:38:59.514384 18803 consensus_queue.cc:260] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:38:59.514430 18804 raft_consensus.cc:348] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:38:59.514648 18804 raft_consensus.cc:374] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:38:59.514699 18804 raft_consensus.cc:729] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4437d75272114d0e85dec0cfcf851f3f, State: Initialized, Role: FOLLOWER
I20260923 14:38:59.515028 18804 consensus_queue.cc:260] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:38:59.515224 18803 ts_tablet_manager.cc:1428] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:38:59.515479 18517 heartbeater.cc:503] Master 127.2.218.62:33485 was elected leader, sending a full tablet report...
I20260923 14:38:59.515491 18802 ts_tablet_manager.cc:1428] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260923 14:38:59.515803 18804 ts_tablet_manager.cc:1428] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:38:59.516091 18785 heartbeater.cc:503] Master 127.2.218.62:33485 was elected leader, sending a full tablet report...
W20260923 14:38:59.573019 18518 tablet.cc:2369] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:38:59.586692 18809 raft_consensus.cc:482] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:38:59.586826 18809 raft_consensus.cc:504] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:38:59.587639 18809 leader_election.cc:288] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801), 391cf02b500543258d46f14cec926d29 (127.2.218.2:43593)
I20260923 14:38:59.590955 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.587810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:43593 (local address 127.2.218.1:40447)
0923 14:38:59.588185 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:59.588197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:59.588295 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:59.588541 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:59.588543 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:59.588556 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:59.588666 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.588672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.589275 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.589277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.589981 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.589989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.590576 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.590579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.590664 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.590684 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:38:59.590744 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 14:38:59.590813 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":73,"thread_start_us":46,"threads_started":1}
I20260923 14:38:59.590950 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.587885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:40447 (local address 127.2.218.2:43593)
0923 14:38:59.587986 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:59.587990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:59.588279 (+ 289us) server_negotiation.cc:407] Connection header received
0923 14:38:59.588379 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:59.588381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:59.588439 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:59.588510 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:59.588759 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.589166 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.590089 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.590461 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.590532 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.590607 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:59.590778 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 14:38:59.590851 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:38:59.590983 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.587932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:42871 (local address 127.2.218.3:32801)
0923 14:38:59.588016 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:59.588019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:59.588164 (+ 145us) server_negotiation.cc:407] Connection header received
0923 14:38:59.588237 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:59.588242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:59.588286 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:59.588356 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:38:59.588648 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.589113 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.590149 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.590484 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.590591 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.590641 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:59.590835 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 14:38:59.590894 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 14:38:59.590987 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.587810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:32801 (local address 127.2.218.1:42871)
0923 14:38:59.588087 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:59.588098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:38:59.588170 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:38:59.588389 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:38:59.588392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:38:59.588404 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:38:59.588550 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.588556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.589208 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.589212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.590007 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:38:59.590016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.590610 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.590614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:38:59.590709 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.590724 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 14:38:59.590808 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 14:38:59.590863 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260923 14:38:59.591622 18605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327ea7091a7043c1a92e789f52c2da6c" candidate_uuid: "185023dcb9a1476a856c03e7b7150b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391cf02b500543258d46f14cec926d29" is_pre_election: true
I20260923 14:38:59.591622 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327ea7091a7043c1a92e789f52c2da6c" candidate_uuid: "185023dcb9a1476a856c03e7b7150b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4437d75272114d0e85dec0cfcf851f3f" is_pre_election: true
I20260923 14:38:59.591857 18605 raft_consensus.cc:2454] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185023dcb9a1476a856c03e7b7150b4e in term 0.
I20260923 14:38:59.591848 18739 raft_consensus.cc:2454] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185023dcb9a1476a856c03e7b7150b4e in term 0.
I20260923 14:38:59.592257 18406 leader_election.cc:302] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 185023dcb9a1476a856c03e7b7150b4e, 391cf02b500543258d46f14cec926d29, 4437d75272114d0e85dec0cfcf851f3f; no voters:
I20260923 14:38:59.592504 18809 raft_consensus.cc:2781] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:38:59.592584 18809 raft_consensus.cc:482] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:38:59.592629 18809 raft_consensus.cc:3037] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:59.593429 18809 raft_consensus.cc:504] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:38:59.593713 18809 leader_election.cc:288] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 1 election: Requested vote from peers 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801), 391cf02b500543258d46f14cec926d29 (127.2.218.2:43593)
I20260923 14:38:59.594027 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327ea7091a7043c1a92e789f52c2da6c" candidate_uuid: "185023dcb9a1476a856c03e7b7150b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4437d75272114d0e85dec0cfcf851f3f"
I20260923 14:38:59.594027 18605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327ea7091a7043c1a92e789f52c2da6c" candidate_uuid: "185023dcb9a1476a856c03e7b7150b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391cf02b500543258d46f14cec926d29"
I20260923 14:38:59.594146 18739 raft_consensus.cc:3037] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:59.594146 18605 raft_consensus.cc:3037] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:38:59.595142 18739 raft_consensus.cc:2454] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185023dcb9a1476a856c03e7b7150b4e in term 1.
I20260923 14:38:59.595142 18605 raft_consensus.cc:2454] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185023dcb9a1476a856c03e7b7150b4e in term 1.
I20260923 14:38:59.595386 18406 leader_election.cc:302] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185023dcb9a1476a856c03e7b7150b4e, 391cf02b500543258d46f14cec926d29; no voters:
I20260923 14:38:59.595553 18809 raft_consensus.cc:2781] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:38:59.595737 18809 raft_consensus.cc:686] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 1 LEADER]: Becoming Leader. State: Replica: 185023dcb9a1476a856c03e7b7150b4e, State: Running, Role: LEADER
I20260923 14:38:59.595989 18809 consensus_queue.cc:237] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:38:59.598313 18324 catalog_manager.cc:5495] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 185023dcb9a1476a856c03e7b7150b4e (127.2.218.1). New cstate: current_term: 1 leader_uuid: "185023dcb9a1476a856c03e7b7150b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } health_report { overall_health: UNKNOWN } } }
I20260923 14:38:59.613485 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.610344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.2.218.1:38711)
0923 14:38:59.610439 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0923 14:38:59.610445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:38:59.610488 (+ 43us) server_negotiation.cc:407] Connection header received
0923 14:38:59.610653 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:38:59.610656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:38:59.610719 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:38:59.610828 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:38:59.611078 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.611528 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.612306 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:38:59.612504 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:38:59.612600 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:38:59.612868 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:38:59.613165 (+ 297us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:38:59.613281 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:38:59.613336 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 14:38:59.613378 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:38:59.618085 18739 raft_consensus.cc:1261] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f [term 1 FOLLOWER]: Refusing update from remote peer 185023dcb9a1476a856c03e7b7150b4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:38:59.618085 18605 raft_consensus.cc:1261] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Refusing update from remote peer 185023dcb9a1476a856c03e7b7150b4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:38:59.618781 18813 consensus_queue.cc:1035] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:38:59.618952 18809 consensus_queue.cc:1035] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:38:59.624348 18816 mvcc.cc:205] Tried to move back new op lower bound from 7332554095069671424 to 7332554094987063296. Current Snapshot: MvccSnapshot[applied={T|T < 7332554095069671424}]
I20260923 14:38:59.625087 18818 mvcc.cc:205] Tried to move back new op lower bound from 7332554095069671424 to 7332554094987063296. Current Snapshot: MvccSnapshot[applied={T|T < 7332554095069671424}]
I20260923 14:38:59.626781 18817 mvcc.cc:205] Tried to move back new op lower bound from 7332554095069671424 to 7332554094987063296. Current Snapshot: MvccSnapshot[applied={T|T < 7332554095069671424}]
I20260923 14:38:59.632264 18325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:38:59.632373 18325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:38:59.634366 18325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Sending DeleteTablet for 3 replicas of tablet 327ea7091a7043c1a92e789f52c2da6c
I20260923 14:38:59.635015 18719 tablet_service.cc:1514] Processing DeleteTablet for tablet 327ea7091a7043c1a92e789f52c2da6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:59 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:39596
I20260923 14:38:59.635073 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet 327ea7091a7043c1a92e789f52c2da6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:59 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:36582
I20260923 14:38:59.635288 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 327ea7091a7043c1a92e789f52c2da6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:59 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:41006
I20260923 14:38:59.635438 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 18388
I20260923 14:38:59.635471 18827 tablet_replica.cc:330] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: stopping tablet replica
I20260923 14:38:59.635746 18827 raft_consensus.cc:2229] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:38:59.635979 18827 raft_consensus.cc:2258] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:38:59.636149 18828 tablet_replica.cc:330] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: stopping tablet replica
I20260923 14:38:59.636451 18828 raft_consensus.cc:2229] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:38:59.636730 18828 raft_consensus.cc:2258] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:38:59.637809 18828 ts_tablet_manager.cc:1905] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:38:59.637809 18827 ts_tablet_manager.cc:1905] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:38:59.640067 18828 ts_tablet_manager.cc:1918] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:38:59.640067 18827 ts_tablet_manager.cc:1918] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:38:59.640147 18827 log.cc:1196] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/327ea7091a7043c1a92e789f52c2da6c
I20260923 14:38:59.640208 18828 log.cc:1196] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/327ea7091a7043c1a92e789f52c2da6c
I20260923 14:38:59.640453 18827 ts_tablet_manager.cc:1939] T 327ea7091a7043c1a92e789f52c2da6c P 4437d75272114d0e85dec0cfcf851f3f: Deleting consensus metadata
I20260923 14:38:59.640484 18828 ts_tablet_manager.cc:1939] T 327ea7091a7043c1a92e789f52c2da6c P 391cf02b500543258d46f14cec926d29: Deleting consensus metadata
I20260923 14:38:59.641443 18312 catalog_manager.cc:4841] TS 391cf02b500543258d46f14cec926d29 (127.2.218.2:43593): tablet 327ea7091a7043c1a92e789f52c2da6c (table test-table [id=d349bbd7a3d5492d8a20e5f747c33763]) successfully deleted
W20260923 14:38:59.641752 18313 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:38711 (error 108)
I20260923 14:38:59.641968 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 18522
I20260923 14:38:59.642162 18313 catalog_manager.cc:4841] TS 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801): tablet 327ea7091a7043c1a92e789f52c2da6c (table test-table [id=d349bbd7a3d5492d8a20e5f747c33763]) successfully deleted
I20260923 14:38:59.643033 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:38:59.642815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:38711 (local address 127.0.0.1:36602)
0923 14:38:59.642927 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 14:38:59.642983 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:38711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 14:38:59.643146 18313 catalog_manager.cc:4584] TS 185023dcb9a1476a856c03e7b7150b4e (127.2.218.1:38711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 327ea7091a7043c1a92e789f52c2da6c: Network error: Client connection negotiation failed: client connection to 127.2.218.1:38711: connect: Connection refused (error 111)
I20260923 14:38:59.647294 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 18656
I20260923 14:38:59.652009 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 18294
I20260923 14:39:19.658393 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33485
--webserver_interface=127.2.218.62
--webserver_port=39597
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:33485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:39:19.760702 18830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:19.760897 18830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:19.760933 18830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:19.764465 18830 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 14:39:19.764511 18830 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 14:39:19.764528 18830 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 14:39:19.764580 18830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:39:19.764611 18830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:19.764647 18830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:19.764695 18830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:39:19.764744 18830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:39:19.769274 18830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:33485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=39597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:19.770100 18830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:19.770973 18830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:19.782450 18836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:19.782554 18830 server_base.cc:1034] running on GCE node
W20260923 14:39:19.782491 18839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:19.782491 18837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:19.783203 18830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:19.784124 18830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:19.785329 18830 hybrid_clock.cc:648] HybridClock initialized: now 1790174359785303 us; error 36 us; skew 500 ppm
Sep 23 14:39:19 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:19.787806 18830 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:39:19.788477 18830 webserver.cc:460] Webserver started at http://127.2.218.62:39597/ using document root <none> and password file <none>
I20260923 14:39:19.788949 18830 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:19.789037 18830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:19.792359 18830 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:39:19.794181 18846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:19.795125 18830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:19.795320 18830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "96a52d6c518b4af0a1e5f194c2760b0b"
format_stamp: "Formatted at 2026-09-23 14:38:58 on dist-test-slave-zl47"
server_key: "9a890ecf853ba7d8fcc056c3301c5b75"
server_key_iv: "ddd578590bb847894cae7bc2fd817b37"
server_key_version: "encryptionkey@0"
I20260923 14:39:19.795845 18830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:19.818472 18830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:19.820914 18830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:19.821107 18830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:19.827702 18830 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:33485
I20260923 14:39:19.827975 18898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:33485 every 8 connection(s)
I20260923 14:39:19.828679 18830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:39:19.832768 18899 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:39:19.834236 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 18830
I20260923 14:39:19.834277 18899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Bootstrap starting.
I20260923 14:39:19.834652 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:38711
--local_ip_for_outbound_sockets=127.2.218.1
--tserver_master_addrs=127.2.218.62:33485
--webserver_port=38295
--webserver_interface=127.2.218.1
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:39:19.850783 18899 log.cc:824] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:19.859282 18899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:39:19.859652 18899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Bootstrap complete.
I20260923 14:39:19.863444 18899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:39:19.863837 18899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a52d6c518b4af0a1e5f194c2760b0b, State: Initialized, Role: FOLLOWER
I20260923 14:39:19.864315 18899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:39:19.864426 18899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:39:19.864513 18899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:39:19.864615 18899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:39:19.866531 18899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:39:19.866868 18899 leader_election.cc:302] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a52d6c518b4af0a1e5f194c2760b0b; no voters:
I20260923 14:39:19.867182 18899 leader_election.cc:288] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 14:39:19.867257 18903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 2 FOLLOWER]: Leader election won for term 2
I20260923 14:39:19.867529 18903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 2 LEADER]: Becoming Leader. State: Replica: 96a52d6c518b4af0a1e5f194c2760b0b, State: Running, Role: LEADER
I20260923 14:39:19.867820 18903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:39:19.868376 18899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:39:19.869405 18905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a52d6c518b4af0a1e5f194c2760b0b. Latest consensus state: current_term: 2 leader_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } } }
I20260923 14:39:19.869426 18904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } } }
I20260923 14:39:19.869565 18904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: This master's current role is: LEADER
I20260923 14:39:19.869566 18905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: This master's current role is: LEADER
I20260923 14:39:19.870199 18912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:39:19.875227 18912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:39:19.875893 18912 catalog_manager.cc:1259] Loaded cluster ID: 97ce71c1ff5f42bab62d07db9a080615
I20260923 14:39:19.875959 18912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:39:19.878217 18912 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:39:19.878846 18912 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Loaded TSK: 0
I20260923 14:39:19.899374 18912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Generated new TSK 1
I20260923 14:39:19.899580 18912 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:39:19.938830 18901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:19.939033 18901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:19.939100 18901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:19.943046 18901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:19.943117 18901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:19.943192 18901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:39:19.947743 18901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:38711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=38295
--tserver_master_addrs=127.2.218.62:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:19.948678 18901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:19.949599 18901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:19.958748 18926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:19.958993 18901 server_base.cc:1034] running on GCE node
W20260923 14:39:19.958882 18929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:19.958745 18927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:19.959579 18901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:19.960202 18901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:19.961382 18901 hybrid_clock.cc:648] HybridClock initialized: now 1790174359961357 us; error 38 us; skew 500 ppm
Sep 23 14:39:19 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:19.968504 18901 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:39:19.969321 18901 webserver.cc:460] Webserver started at http://127.2.218.1:38295/ using document root <none> and password file <none>
I20260923 14:39:19.969909 18901 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:19.970003 18901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:19.973376 18901 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:19.975445 18936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:19.976500 18901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:39:19.976611 18901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "185023dcb9a1476a856c03e7b7150b4e"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "8c2b2c4e409da0b2ce4d1b01019982e5"
server_key_iv: "5f0e9572c12b169f7fb07b4636bbb42b"
server_key_version: "encryptionkey@0"
I20260923 14:39:19.977038 18901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:20.005097 18901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:20.008105 18901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:20.008287 18901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:20.008821 18901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:20.009887 18943 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 14:39:20.014075 18901 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 14:39:20.014173 18901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260923 14:39:20.014235 18901 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 14:39:20.015683 18901 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 14:39:20.015733 18901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260923 14:39:20.015982 18943 tablet_bootstrap.cc:492] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Bootstrap starting.
I20260923 14:39:20.026036 18901 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:38711
I20260923 14:39:20.026149 19050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:38711 every 8 connection(s)
I20260923 14:39:20.027235 18901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:39:20.030736 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 18901
I20260923 14:39:20.031245 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:43593
--local_ip_for_outbound_sockets=127.2.218.2
--tserver_master_addrs=127.2.218.62:33485
--webserver_port=33079
--webserver_interface=127.2.218.2
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:39:20 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:39:20.039652 18943 log.cc:824] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:20.045733 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.029255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:50737)
0923 14:39:20.029788 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:20.029830 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:20.030745 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:20.031989 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:20.032000 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:20.032472 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:20.032982 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.032995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.034906 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.034910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.036258 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.036271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.037297 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.037303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.037526 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.038152 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:39:20.038174 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:20.040092 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:20.042743 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:20.042752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:20.042767 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:20.043164 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:20.043574 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:20.043578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:20.043582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:20.043723 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:20.044330 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:20.044336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:20.044560 (+ 224us) client_negotiation.cc:769] Sending connection context
0923 14:39:20.044721 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 14:39:20.044984 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":137,"threads_started":1}
I20260923 14:39:20.045928 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.029268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:50737 (local address 127.2.218.62:33485)
0923 14:39:20.029623 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.029643 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.029941 (+ 298us) server_negotiation.cc:407] Connection header received
0923 14:39:20.031219 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.031233 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.031578 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.031910 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:20.033152 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.034746 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.036409 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.037142 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.037314 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.040299 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:20.040307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:20.040322 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:20.040354 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:20.042597 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.043308 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.043315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.043322 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.043465 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.043817 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.043820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.043821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.044179 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:20.044282 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.044853 (+ 571us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.045108 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":122,"threads_started":1}
I20260923 14:39:20.047955 19051 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33485
I20260923 14:39:20.048167 19051 heartbeater.cc:464] Registering TS with master...
I20260923 14:39:20.048264 18943 tablet_bootstrap.cc:492] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:39:20.048636 18943 tablet_bootstrap.cc:492] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Bootstrap complete.
I20260923 14:39:20.048652 19051 heartbeater.cc:511] Master 127.2.218.62:33485 requested a full tablet report, sending...
I20260923 14:39:20.049813 18943 ts_tablet_manager.cc:1397] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.000s
I20260923 14:39:20.051066 18863 ts_manager.cc:194] Registered new tserver with Master: 185023dcb9a1476a856c03e7b7150b4e (127.2.218.1:38711)
I20260923 14:39:20.052213 18943 raft_consensus.cc:348] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:39:20.052624 18943 raft_consensus.cc:729] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185023dcb9a1476a856c03e7b7150b4e, State: Initialized, Role: FOLLOWER
I20260923 14:39:20.053025 18943 consensus_queue.cc:260] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } }
I20260923 14:39:20.053840 18943 ts_tablet_manager.cc:1428] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260923 14:39:20.054713 18863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:50737
I20260923 14:39:20.057420 19051 heartbeater.cc:503] Master 127.2.218.62:33485 was elected leader, sending a full tablet report...
Sep 23 14:39:20 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174359, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:39:20.063180 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.054220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.2.218.1:38711)
0923 14:39:20.054840 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.054863 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.054883 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:39:20.054939 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.054942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.055089 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.055214 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:20.055886 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.056520 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.057281 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.057637 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.057752 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.059804 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:20.059812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:20.059820 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:20.059838 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:20.061241 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.061832 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.061837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.061838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.061919 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.062264 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.062266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.062267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.062473 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:20.062570 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.062958 (+ 388us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.063073 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":94,"threads_started":1}
I20260923 14:39:20.063236 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.054090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:38711 (local address 127.0.0.1:54594)
0923 14:39:20.054435 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:20.054472 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:20.054574 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:20.055219 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:20.055224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:20.055667 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:20.055777 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.055783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.056622 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.056626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.057174 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.057182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.057755 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.057759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.057860 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.058352 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:39:20.058364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:20.059663 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:20.061348 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:20.061356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:20.061378 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:20.061721 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:20.062012 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:20.062020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:20.062025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:20.062162 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:20.062590 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:20.062596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:20.062719 (+ 123us) client_negotiation.cc:769] Sending connection context
0923 14:39:20.062878 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 14:39:20.063057 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
I20260923 14:39:20.064286 18985 tablet_service.cc:1514] Processing DeleteTablet for tablet 327ea7091a7043c1a92e789f52c2da6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:38:59 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:54594
I20260923 14:39:20.064949 19060 tablet_replica.cc:330] stopping tablet replica
I20260923 14:39:20.065186 19060 raft_consensus.cc:2229] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:39:20.065383 19060 raft_consensus.cc:2258] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:39:20.066206 19060 ts_tablet_manager.cc:1905] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:39:20.069482 19060 ts_tablet_manager.cc:1918] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:39:20.069554 19060 log.cc:1196] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal/wals/327ea7091a7043c1a92e789f52c2da6c
I20260923 14:39:20.069856 19060 ts_tablet_manager.cc:1939] T 327ea7091a7043c1a92e789f52c2da6c P 185023dcb9a1476a856c03e7b7150b4e: Deleting consensus metadata
I20260923 14:39:20.071388 18850 catalog_manager.cc:4841] TS 185023dcb9a1476a856c03e7b7150b4e (127.2.218.1:38711): tablet 327ea7091a7043c1a92e789f52c2da6c (table test-table [id=d349bbd7a3d5492d8a20e5f747c33763]) successfully deleted
W20260923 14:39:20.141324 19055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:20.141548 19055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:20.141620 19055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:20.145339 19055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:20.145416 19055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:20.145498 19055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:39:20.149964 19055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:43593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=33079
--tserver_master_addrs=127.2.218.62:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:20.150856 19055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:20.151690 19055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:20.160328 19066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:20.160365 19065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:20.160338 19068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:20.160670 19055 server_base.cc:1034] running on GCE node
I20260923 14:39:20.161010 19055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:20.161522 19055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:20.162672 19055 hybrid_clock.cc:648] HybridClock initialized: now 1790174360162655 us; error 31 us; skew 500 ppm
Sep 23 14:39:20 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:20.165076 19055 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:39:20.165788 19055 webserver.cc:460] Webserver started at http://127.2.218.2:33079/ using document root <none> and password file <none>
I20260923 14:39:20.166281 19055 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:20.166374 19055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:20.169822 19055 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:39:20.171761 19075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:20.172890 19055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:39:20.172979 19055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "391cf02b500543258d46f14cec926d29"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "953e60b06c1883677999e787ca4e45c2"
server_key_iv: "07dcaaacd38e84f175d722367edf4f2a"
server_key_version: "encryptionkey@0"
I20260923 14:39:20.173422 19055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:20.205163 19055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:20.207860 19055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:20.207995 19055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:20.208449 19055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:20.209111 19055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:39:20.209151 19055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:20.209184 19055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:39:20.209200 19055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:20.216771 19055 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:43593
I20260923 14:39:20.216796 19188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:43593 every 8 connection(s)
I20260923 14:39:20.217638 19055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:39:20.217876 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 19055
I20260923 14:39:20.218392 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:32801
--local_ip_for_outbound_sockets=127.2.218.3
--tserver_master_addrs=127.2.218.62:33485
--webserver_port=37239
--webserver_interface=127.2.218.3
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 14:39:20 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:39:20.230965 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.219723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:33069 (local address 127.2.218.62:33485)
0923 14:39:20.219841 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.219845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.220694 (+ 849us) server_negotiation.cc:407] Connection header received
0923 14:39:20.221486 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.221489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.221548 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.221631 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:20.222680 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.223526 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.224661 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.225122 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.225261 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.227873 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:20.227876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:20.227877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:20.227901 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:20.229130 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.229668 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.229670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.229671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.229738 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.230050 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.230052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.230052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.230243 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:20.230312 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.230757 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.230852 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 14:39:20.231824 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.219974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:33069)
0923 14:39:20.220560 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:20.220598 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:20.221293 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:20.221814 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:20.221820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:20.222158 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:20.222538 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.222549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.223656 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.223659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.224518 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.224527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.225268 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.225270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.225426 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.225966 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:39:20.225982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:20.227716 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:20.229234 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:20.229241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:20.229254 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:20.229577 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:20.229815 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:20.229818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:20.229820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:20.229971 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:20.230323 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:20.230328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:20.230559 (+ 231us) client_negotiation.cc:769] Sending connection context
0923 14:39:20.230724 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 14:39:20.230981 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":132,"threads_started":1}
I20260923 14:39:20.232970 19189 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33485
I20260923 14:39:20.233186 19189 heartbeater.cc:464] Registering TS with master...
I20260923 14:39:20.233628 19189 heartbeater.cc:511] Master 127.2.218.62:33485 requested a full tablet report, sending...
I20260923 14:39:20.234592 18863 ts_manager.cc:194] Registered new tserver with Master: 391cf02b500543258d46f14cec926d29 (127.2.218.2:43593)
I20260923 14:39:20.235096 18863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:33069
W20260923 14:39:20.323848 19191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:20.324041 19191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:20.324110 19191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:20.327808 19191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:20.327853 19191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:20.327950 19191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:39:20.332559 19191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:32801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=37239
--tserver_master_addrs=127.2.218.62:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:20.333304 19191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:20.334138 19191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:20.342361 19198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:20.342422 19200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:20.342478 19191 server_base.cc:1034] running on GCE node
W20260923 14:39:20.342425 19197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:20.342900 19191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:20.343478 19191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:20.344645 19191 hybrid_clock.cc:648] HybridClock initialized: now 1790174360344634 us; error 23 us; skew 500 ppm
Sep 23 14:39:20 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:20.347048 19191 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:39:20.347765 19191 webserver.cc:460] Webserver started at http://127.2.218.3:37239/ using document root <none> and password file <none>
I20260923 14:39:20.348232 19191 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:20.348323 19191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:20.351560 19191 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:39:20.353263 19207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:20.354370 19191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:20.354477 19191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "4437d75272114d0e85dec0cfcf851f3f"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "3a30f4ae3cbb577c8967184b5a7356c4"
server_key_iv: "8d66a243f6bf96191aaa9678fb335ca6"
server_key_version: "encryptionkey@0"
I20260923 14:39:20.355019 19191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:20.384605 19191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:20.386966 19191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:20.387147 19191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:20.387617 19191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:20.388280 19191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:39:20.388350 19191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:20.388427 19191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:39:20.388471 19191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:20.396406 19191 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:32801
I20260923 14:39:20.396442 19320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:32801 every 8 connection(s)
I20260923 14:39:20.397239 19191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:39:20.404978 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 19191
Sep 23 14:39:20 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174360, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:39:20.409875 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.398801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:37531 (local address 127.2.218.62:33485)
0923 14:39:20.398924 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.398928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.400084 (+ 1156us) server_negotiation.cc:407] Connection header received
0923 14:39:20.400852 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.400855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.400908 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.401002 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:20.401974 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.402546 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.403615 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.403839 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.403942 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.406323 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:20.406328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:20.406333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:20.406356 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:20.407961 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.408524 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.408526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.408527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.408599 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.408931 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.408933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.408934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.409110 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:20.409199 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.409625 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.409711 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:39:20.410789 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.399136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:37531)
0923 14:39:20.399959 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:20.400007 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:20.400673 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:20.401159 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:20.401165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:20.401459 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:20.401844 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.401856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.402667 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.402670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.403494 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.403504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.403979 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.403987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.404152 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.404573 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:39:20.404588 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:20.406179 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:20.408095 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:20.408100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:20.408113 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:20.408434 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:20.408682 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:20.408685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:20.408687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:20.408840 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:20.409206 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:20.409210 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:20.409391 (+ 181us) client_negotiation.cc:769] Sending connection context
0923 14:39:20.409588 (+ 197us) client_negotiation.cc:240] Negotiation successful
0923 14:39:20.409880 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":492,"threads_started":1}
I20260923 14:39:20.411856 19321 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33485
I20260923 14:39:20.412071 19321 heartbeater.cc:464] Registering TS with master...
I20260923 14:39:20.412608 19321 heartbeater.cc:511] Master 127.2.218.62:33485 requested a full tablet report, sending...
I20260923 14:39:20.413671 18863 ts_manager.cc:194] Registered new tserver with Master: 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801)
I20260923 14:39:20.414378 18863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:37531
I20260923 14:39:20.414418 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.405577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.2.218.62:33485)
0923 14:39:20.405810 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.405814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.406009 (+ 195us) server_negotiation.cc:407] Connection header received
0923 14:39:20.406199 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.406201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.406240 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.406363 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:20.407161 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.407967 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.409106 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.409380 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.409450 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.411206 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:20.411210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:20.411212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:20.411237 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:20.412859 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.413283 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.413285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.413286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.413388 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.413644 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.413646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.413647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.413847 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:20.413992 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.414196 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.414275 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260923 14:39:20.416064 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:39:20.419020 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.416591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.2.218.62:33485)
0923 14:39:20.416682 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.416685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.416821 (+ 136us) server_negotiation.cc:407] Connection header received
0923 14:39:20.416920 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.416922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.416990 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.417062 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:39:20.417192 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.417569 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.418259 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.418441 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.418505 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.418704 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:39:20.418792 (+ 88us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 14:39:20.418954 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46320: token expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260923 14:39:20.419080 19324 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46320: token expired
W20260923 14:39:20.419348 18789 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.218.62:33485)
I20260923 14:39:20.424642 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.419639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.2.218.62:33485)
0923 14:39:20.419725 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.419728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.419752 (+ 24us) server_negotiation.cc:407] Connection header received
0923 14:39:20.419810 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.419812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.419847 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.419895 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:20.420327 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.420717 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.421328 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.421491 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.421553 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.422507 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:20.422510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:20.422511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:20.422528 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:20.423676 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.423956 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.423958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.423959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.424019 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:20.424181 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:20.424182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:20.424183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:20.424294 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:20.424365 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.424499 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.424564 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 14:39:20.426378 18863 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:39:20.429014 18863 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:39:20.435084 18985 tablet_service.cc:1467] Processing CreateTablet for tablet b2334dddcae1456e8b40c214fec4005f (DEFAULT_TABLE table=test-table [id=bbb5a052539145879c6a2f530c5a6020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:39:20.435635 18985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2334dddcae1456e8b40c214fec4005f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:20.438127 18943 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Bootstrap starting.
I20260923 14:39:20.438580 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.434436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:43593 (local address 127.0.0.1:42304)
0923 14:39:20.434666 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:20.434677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:20.434749 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:20.435337 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:20.435339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:20.435369 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:20.435455 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.435460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.436473 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.436476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.437447 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.437455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.438249 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.438252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.438326 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.438365 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 14:39:20.438414 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 14:39:20.438470 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"spinlock_wait_cycles":9984,"thread_start_us":71,"threads_started":1}
I20260923 14:39:20.438791 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.434133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:32801 (local address 127.0.0.1:47464)
0923 14:39:20.434227 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:20.434240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:20.434297 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:20.435384 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:20.435386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:20.435399 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:20.435456 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.435460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.436528 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.436532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.437492 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.437501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.438454 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.438457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.438535 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.438552 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:39:20.438613 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 14:39:20.438681 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"spinlock_wait_cycles":384}
I20260923 14:39:20.438894 18943 tablet_bootstrap.cc:654] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:20.438928 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.434436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.2.218.3:32801)
0923 14:39:20.435001 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.435010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.435027 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:39:20.435077 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.435081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.435231 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.435393 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:20.435560 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.436390 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.437622 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.438314 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.438452 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.438561 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.438636 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.438801 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":128,"threads_started":1}
I20260923 14:39:20.439767 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.434669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.2.218.2:43593)
0923 14:39:20.434976 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.434984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.435001 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:39:20.435054 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.435057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.435178 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.435599 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:20.435768 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.436365 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.437562 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.438099 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.438744 (+ 645us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.438902 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.438949 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.439031 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":139,"threads_started":1}
I20260923 14:39:20.440680 18943 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: No bootstrap required, opened a new log
I20260923 14:39:20.440747 18943 ts_tablet_manager.cc:1397] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 14:39:20.441133 18943 raft_consensus.cc:348] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:20.441228 18943 raft_consensus.cc:374] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:20.440979 19255 tablet_service.cc:1467] Processing CreateTablet for tablet b2334dddcae1456e8b40c214fec4005f (DEFAULT_TABLE table=test-table [id=bbb5a052539145879c6a2f530c5a6020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:39:20.441262 18943 raft_consensus.cc:729] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185023dcb9a1476a856c03e7b7150b4e, State: Initialized, Role: FOLLOWER
I20260923 14:39:20.441382 18943 consensus_queue.cc:260] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:20.441608 18943 ts_tablet_manager.cc:1428] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 14:39:20.441754 19255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2334dddcae1456e8b40c214fec4005f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:20.442128 19123 tablet_service.cc:1467] Processing CreateTablet for tablet b2334dddcae1456e8b40c214fec4005f (DEFAULT_TABLE table=test-table [id=bbb5a052539145879c6a2f530c5a6020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:39:20.443022 19123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2334dddcae1456e8b40c214fec4005f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:20.447484 19334 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Bootstrap starting.
I20260923 14:39:20.448424 19335 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Bootstrap starting.
I20260923 14:39:20.449527 19334 tablet_bootstrap.cc:654] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:20.450289 19334 log.cc:824] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:20.450402 19335 tablet_bootstrap.cc:654] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:20.450686 19057 raft_consensus.cc:482] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:39:20.450805 19057 raft_consensus.cc:504] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:20.451112 19335 log.cc:824] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:20.451728 19057 leader_election.cc:288] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801), 391cf02b500543258d46f14cec926d29 (127.2.218.2:43593)
I20260923 14:39:20.451924 19334 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: No bootstrap required, opened a new log
I20260923 14:39:20.452081 19334 ts_tablet_manager.cc:1397] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:39:20.453145 19335 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: No bootstrap required, opened a new log
I20260923 14:39:20.453310 19335 ts_tablet_manager.cc:1397] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260923 14:39:20.455456 19334 raft_consensus.cc:348] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:20.455704 19334 raft_consensus.cc:374] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:20.455763 19334 raft_consensus.cc:729] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4437d75272114d0e85dec0cfcf851f3f, State: Initialized, Role: FOLLOWER
I20260923 14:39:20.455906 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.452017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:35881 (local address 127.2.218.2:43593)
0923 14:39:20.452102 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.452105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.452294 (+ 189us) server_negotiation.cc:407] Connection header received
0923 14:39:20.452371 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.452374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.452411 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.452482 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:20.452716 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.453322 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.454704 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.455252 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.455324 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.455368 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.455736 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.455806 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 14:39:20.456255 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.451959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:43593 (local address 127.2.218.1:35881)
0923 14:39:20.452215 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:20.452227 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:20.452303 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:20.452500 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:20.452503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:20.452517 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:20.452619 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.452625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.453447 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.453450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.454578 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.454589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.455478 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.455482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.455574 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.455617 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 14:39:20.456068 (+ 451us) client_negotiation.cc:240] Negotiation successful
0923 14:39:20.456135 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260923 14:39:20.456197 19334 consensus_queue.cc:260] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:20.456451 19335 raft_consensus.cc:348] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:20.456677 19335 raft_consensus.cc:374] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:20.456732 19335 raft_consensus.cc:729] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391cf02b500543258d46f14cec926d29, State: Initialized, Role: FOLLOWER
I20260923 14:39:20.456844 19143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2334dddcae1456e8b40c214fec4005f" candidate_uuid: "185023dcb9a1476a856c03e7b7150b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391cf02b500543258d46f14cec926d29" is_pre_election: true
I20260923 14:39:20.457021 19334 ts_tablet_manager.cc:1428] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260923 14:39:20.457084 19335 consensus_queue.cc:260] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:20.457186 19321 heartbeater.cc:503] Master 127.2.218.62:33485 was elected leader, sending a full tablet report...
I20260923 14:39:20.457597 19189 heartbeater.cc:503] Master 127.2.218.62:33485 was elected leader, sending a full tablet report...
I20260923 14:39:20.457828 19335 ts_tablet_manager.cc:1428] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20260923 14:39:20.458657 19143 raft_consensus.cc:2454] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185023dcb9a1476a856c03e7b7150b4e in term 0.
I20260923 14:39:20.458912 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.451837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:32801 (local address 127.2.218.1:50463)
0923 14:39:20.452069 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:20.452080 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:20.452152 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:20.455073 (+ 2921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:20.455076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:20.455093 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:20.455186 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.455191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.456821 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.456824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.457737 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:20.457747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.458541 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.458544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:20.458637 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.458654 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 14:39:20.458714 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 14:39:20.458775 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260923 14:39:20.459411 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.451961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:50463 (local address 127.2.218.3:32801)
0923 14:39:20.454199 (+ 2238us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.454202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.454214 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:39:20.454248 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.454251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.454283 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.455278 (+ 995us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:20.455351 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.456272 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.458078 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.458435 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.458531 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.458570 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.459143 (+ 573us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.459287 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2178}
I20260923 14:39:20.459483 18939 leader_election.cc:302] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185023dcb9a1476a856c03e7b7150b4e, 391cf02b500543258d46f14cec926d29; no voters:
I20260923 14:39:20.459728 19057 raft_consensus.cc:2781] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:39:20.459766 19275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2334dddcae1456e8b40c214fec4005f" candidate_uuid: "185023dcb9a1476a856c03e7b7150b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4437d75272114d0e85dec0cfcf851f3f" is_pre_election: true
I20260923 14:39:20.459821 19057 raft_consensus.cc:482] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:39:20.459885 19057 raft_consensus.cc:3037] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:20.459960 19275 raft_consensus.cc:2454] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185023dcb9a1476a856c03e7b7150b4e in term 0.
I20260923 14:39:20.460732 19057 raft_consensus.cc:504] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:20.461016 19057 leader_election.cc:288] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 1 election: Requested vote from peers 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801), 391cf02b500543258d46f14cec926d29 (127.2.218.2:43593)
I20260923 14:39:20.461252 19275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2334dddcae1456e8b40c214fec4005f" candidate_uuid: "185023dcb9a1476a856c03e7b7150b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4437d75272114d0e85dec0cfcf851f3f"
I20260923 14:39:20.461282 19143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2334dddcae1456e8b40c214fec4005f" candidate_uuid: "185023dcb9a1476a856c03e7b7150b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391cf02b500543258d46f14cec926d29"
I20260923 14:39:20.461355 19275 raft_consensus.cc:3037] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:20.461383 19143 raft_consensus.cc:3037] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:20.462448 19143 raft_consensus.cc:2454] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185023dcb9a1476a856c03e7b7150b4e in term 1.
I20260923 14:39:20.462448 19275 raft_consensus.cc:2454] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185023dcb9a1476a856c03e7b7150b4e in term 1.
I20260923 14:39:20.462759 18939 leader_election.cc:302] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185023dcb9a1476a856c03e7b7150b4e, 391cf02b500543258d46f14cec926d29; no voters:
I20260923 14:39:20.462918 19057 raft_consensus.cc:2781] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:39:20.463133 19057 raft_consensus.cc:686] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 1 LEADER]: Becoming Leader. State: Replica: 185023dcb9a1476a856c03e7b7150b4e, State: Running, Role: LEADER
I20260923 14:39:20.463379 19057 consensus_queue.cc:237] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:20.465478 18863 catalog_manager.cc:5495] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 185023dcb9a1476a856c03e7b7150b4e (127.2.218.1). New cstate: current_term: 1 leader_uuid: "185023dcb9a1476a856c03e7b7150b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } health_report { overall_health: HEALTHY } } }
W20260923 14:39:20.470222 19190 tablet.cc:2369] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:39:20.474315 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.471880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.2.218.1:38711)
0923 14:39:20.471972 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:20.471975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:20.471992 (+ 17us) server_negotiation.cc:407] Connection header received
0923 14:39:20.472076 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:20.472077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:20.472113 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:20.472180 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:39:20.472329 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.472735 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.473353 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:20.473539 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:20.473604 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:20.473771 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:39:20.474043 (+ 272us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:39:20.474154 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:20.474207 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 14:39:20.474242 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:39:20.477144 19275 raft_consensus.cc:1261] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 1 FOLLOWER]: Refusing update from remote peer 185023dcb9a1476a856c03e7b7150b4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:39:20.477144 19143 raft_consensus.cc:1261] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Refusing update from remote peer 185023dcb9a1476a856c03e7b7150b4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:39:20.477708 19057 consensus_queue.cc:1035] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:39:20.477859 19341 consensus_queue.cc:1035] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:39:20.483040 19344 mvcc.cc:205] Tried to move back new op lower bound from 7332554180509433856 to 7332554180459614208. Current Snapshot: MvccSnapshot[applied={T|T < 7332554180509433856}]
I20260923 14:39:20.483644 19343 mvcc.cc:205] Tried to move back new op lower bound from 7332554180509433856 to 7332554180459614208. Current Snapshot: MvccSnapshot[applied={T|T < 7332554180509433856}]
I20260923 14:39:20.489185 18863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:39:20.489332 18863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:39:20.491401 18863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Sending DeleteTablet for 3 replicas of tablet b2334dddcae1456e8b40c214fec4005f
I20260923 14:39:20.491873 19255 tablet_service.cc:1514] Processing DeleteTablet for tablet b2334dddcae1456e8b40c214fec4005f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:39:20 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:47464
I20260923 14:39:20.492017 19123 tablet_service.cc:1514] Processing DeleteTablet for tablet b2334dddcae1456e8b40c214fec4005f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:39:20 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:42304
I20260923 14:39:20.492297 19352 tablet_replica.cc:330] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: stopping tablet replica
I20260923 14:39:20.492497 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 18901
I20260923 14:39:20.492547 19352 raft_consensus.cc:2229] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:39:20.493248 19353 tablet_replica.cc:330] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: stopping tablet replica
I20260923 14:39:20.493345 19352 raft_consensus.cc:2258] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:39:20.493479 19353 raft_consensus.cc:2229] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:39:20.493703 19353 raft_consensus.cc:2258] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:39:20.494665 19352 ts_tablet_manager.cc:1905] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:39:20.497619 19352 ts_tablet_manager.cc:1918] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:39:20.497709 19352 log.cc:1196] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/b2334dddcae1456e8b40c214fec4005f
I20260923 14:39:20.498029 19352 ts_tablet_manager.cc:1939] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Deleting consensus metadata
I20260923 14:39:20.499056 18850 catalog_manager.cc:4841] TS 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801): tablet b2334dddcae1456e8b40c214fec4005f (table test-table [id=bbb5a052539145879c6a2f530c5a6020]) successfully deleted
W20260923 14:39:20.499655 18850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:38711 (error 108)
I20260923 14:39:20.499761 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 19055
I20260923 14:39:20.501029 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.500801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:38711 (local address 127.0.0.1:54612)
0923 14:39:20.500913 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:20.500975 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:38711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 14:39:20.501155 18850 catalog_manager.cc:4584] TS 185023dcb9a1476a856c03e7b7150b4e (127.2.218.1:38711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2334dddcae1456e8b40c214fec4005f: Network error: Client connection negotiation failed: client connection to 127.2.218.1:38711: connect: Connection refused (error 111)
I20260923 14:39:20.504997 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 19191
I20260923 14:39:20.506147 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:20.505103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:43593 (local address 127.0.0.1:42306)
0923 14:39:20.505995 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:20.506053 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.2:43593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":815}
I20260923 14:39:20.509631 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 18830
I20260923 14:39:40.515594 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33485
--webserver_interface=127.2.218.62
--webserver_port=39597
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:33485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:39:40.609244 19355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:40.609442 19355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:40.609475 19355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:40.612929 19355 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 14:39:40.612968 19355 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 14:39:40.612982 19355 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 14:39:40.612993 19355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:39:40.613008 19355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:40.613062 19355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:40.613090 19355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:39:40.613118 19355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:39:40.617349 19355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:33485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=39597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:40.618147 19355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:40.618992 19355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:40.628698 19363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:40.628738 19355 server_base.cc:1034] running on GCE node
W20260923 14:39:40.628703 19360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:40.628703 19361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:40.629527 19355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:40.630438 19355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:40.631587 19355 hybrid_clock.cc:648] HybridClock initialized: now 1790174380631573 us; error 27 us; skew 500 ppm
Sep 23 14:39:40 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174380, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:40.637312 19355 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:39:40.638064 19355 webserver.cc:460] Webserver started at http://127.2.218.62:39597/ using document root <none> and password file <none>
I20260923 14:39:40.638573 19355 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:40.638659 19355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:40.641834 19355 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260923 14:39:40.643877 19370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:40.645008 19355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260923 14:39:40.645159 19355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "96a52d6c518b4af0a1e5f194c2760b0b"
format_stamp: "Formatted at 2026-09-23 14:38:58 on dist-test-slave-zl47"
server_key: "9a890ecf853ba7d8fcc056c3301c5b75"
server_key_iv: "ddd578590bb847894cae7bc2fd817b37"
server_key_version: "encryptionkey@0"
I20260923 14:39:40.645663 19355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:40.658658 19355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:40.661180 19355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:40.661367 19355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:40.667881 19355 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:33485
I20260923 14:39:40.668128 19422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:33485 every 8 connection(s)
I20260923 14:39:40.668897 19355 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:39:40.671051 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 19355
I20260923 14:39:40.671468 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:38711
--local_ip_for_outbound_sockets=127.2.218.1
--tserver_master_addrs=127.2.218.62:33485
--webserver_port=38295
--webserver_interface=127.2.218.1
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:39:40.673170 19423 sys_catalog.cc:263] Verifying existing consensus state
I20260923 14:39:40.675283 19423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Bootstrap starting.
I20260923 14:39:40.697150 19423 log.cc:824] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:40.709903 19423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:39:40.710259 19423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Bootstrap complete.
I20260923 14:39:40.714834 19423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:39:40.715142 19423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a52d6c518b4af0a1e5f194c2760b0b, State: Initialized, Role: FOLLOWER
I20260923 14:39:40.715583 19423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:39:40.715754 19423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:39:40.715849 19423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:39:40.715968 19423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 2 FOLLOWER]: Advancing to term 3
I20260923 14:39:40.717653 19423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:39:40.717978 19423 leader_election.cc:302] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a52d6c518b4af0a1e5f194c2760b0b; no voters:
I20260923 14:39:40.718266 19423 leader_election.cc:288] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 14:39:40.718369 19427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 3 FOLLOWER]: Leader election won for term 3
I20260923 14:39:40.718569 19427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [term 3 LEADER]: Becoming Leader. State: Replica: 96a52d6c518b4af0a1e5f194c2760b0b, State: Running, Role: LEADER
I20260923 14:39:40.718818 19427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } }
I20260923 14:39:40.719281 19423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:39:40.719954 19428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } } }
I20260923 14:39:40.720065 19428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: This master's current role is: LEADER
I20260923 14:39:40.720642 19429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a52d6c518b4af0a1e5f194c2760b0b. Latest consensus state: current_term: 3 leader_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a52d6c518b4af0a1e5f194c2760b0b" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33485 } } }
I20260923 14:39:40.720723 19429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b [sys.catalog]: This master's current role is: LEADER
I20260923 14:39:40.721026 19436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:39:40.725476 19436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:39:40.726065 19436 catalog_manager.cc:1259] Loaded cluster ID: 97ce71c1ff5f42bab62d07db9a080615
I20260923 14:39:40.726154 19436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:39:40.728287 19436 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:39:40.728905 19436 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Loaded TSK: 0
I20260923 14:39:40.728976 19436 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 96a52d6c518b4af0a1e5f194c2760b0b: Loaded TSK: 1
F20260923 14:39:40.729595 19436 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7febcdfc09ad google::LogMessage::Fail() at ??:0
@ 0x7febcdfc388a google::LogMessage::SendToLog() at ??:0
@ 0x7febcdfc04de google::LogMessage::Flush() at ??:0
@ 0x7febcdfc1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7febd0c8418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7febd0c7cf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7febd89257bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7febd88ff99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7febd8900178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7febd8941df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7febd896082b std::function<>::operator()() at ??:0
@ 0x7febd88ffb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7febd8900f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7febd88fd579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7febd8941640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7febd02123d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7febd0212c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7febd02143f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7febd0203512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7febd18476db start_thread at ??:0
@ 0x7febccabe61f clone at ??:0
W20260923 14:39:40.811791 19425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:40.812052 19425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:40.812104 19425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:40.817140 19425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:40.817204 19425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:40.817272 19425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:39:40.823979 19425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:38711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=38295
--tserver_master_addrs=127.2.218.62:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:40.825011 19425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:40.826084 19425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:40.835448 19471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:40.835466 19472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:40.835518 19474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:40.836228 19425 server_base.cc:1034] running on GCE node
I20260923 14:39:40.836596 19425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:40.837189 19425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:40.838376 19425 hybrid_clock.cc:648] HybridClock initialized: now 1790174380838348 us; error 35 us; skew 500 ppm
Sep 23 14:39:40 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174380, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:40.841269 19425 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:39:40.842237 19425 webserver.cc:460] Webserver started at http://127.2.218.1:38295/ using document root <none> and password file <none>
I20260923 14:39:40.842828 19425 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:40.842903 19425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:40.847070 19425 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260923 14:39:40.849371 19481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:40.850673 19425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:39:40.850791 19425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "185023dcb9a1476a856c03e7b7150b4e"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "8c2b2c4e409da0b2ce4d1b01019982e5"
server_key_iv: "5f0e9572c12b169f7fb07b4636bbb42b"
server_key_version: "encryptionkey@0"
I20260923 14:39:40.851387 19425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:40.880264 19425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:40.883554 19425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:40.883742 19425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:40.884331 19425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:40.885402 19488 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 14:39:40.888885 19425 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 14:39:40.888950 19425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260923 14:39:40.889008 19425 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 14:39:40.890815 19425 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 14:39:40.890889 19425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260923 14:39:40.891139 19488 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Bootstrap starting.
I20260923 14:39:40.900524 19425 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:38711
I20260923 14:39:40.900718 19595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:38711 every 8 connection(s)
I20260923 14:39:40.901510 19425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:39:40.908380 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 19425
I20260923 14:39:40.908835 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:43593
--local_ip_for_outbound_sockets=127.2.218.2
--tserver_master_addrs=127.2.218.62:33485
--webserver_port=33079
--webserver_interface=127.2.218.2
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:39:40.910550 19488 log.cc:824] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:40.916628 19488 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:39:40.916941 19488 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Bootstrap complete.
I20260923 14:39:40.917939 19488 ts_tablet_manager.cc:1397] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.008s
I20260923 14:39:40.919943 19488 raft_consensus.cc:348] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:40.920291 19488 raft_consensus.cc:729] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185023dcb9a1476a856c03e7b7150b4e, State: Initialized, Role: FOLLOWER
I20260923 14:39:40.920775 19488 consensus_queue.cc:260] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:40.921586 19488 ts_tablet_manager.cc:1428] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Time spent starting tablet: real 0.004s user 0.006s sys 0.001s
W20260923 14:39:40.958660 19598 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:39:40.903219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:44901)
0923 14:39:40.903635 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:40.903679 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:40.904563 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:40.957986 (+ 53423us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":140,"threads_started":1}
W20260923 14:39:40.959100 19596 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:33485 (0 consecutive failures): Network error: Failed to ping master at 127.2.218.62:33485: Client connection negotiation failed: client connection to 127.2.218.62:33485: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 14:39:40.959740 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:40.959506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:34909)
0923 14:39:40.959622 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:40.959683 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 14:39:40.960395 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:40.960223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:58441)
0923 14:39:40.960318 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:40.960353 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 14:39:40.960563 19596 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 14:39:41.020588 19599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:41.020835 19599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:41.020905 19599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:41.024613 19599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:41.024688 19599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:41.024766 19599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:39:41.029110 19599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:43593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=33079
--tserver_master_addrs=127.2.218.62:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:41.029901 19599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:41.030726 19599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:41.038749 19606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:41.038759 19607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:41.038757 19609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:41.038853 19599 server_base.cc:1034] running on GCE node
I20260923 14:39:41.039319 19599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:41.039840 19599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:41.041036 19599 hybrid_clock.cc:648] HybridClock initialized: now 1790174381041024 us; error 31 us; skew 500 ppm
Sep 23 14:39:41 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:41.043721 19599 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:39:41.044468 19599 webserver.cc:460] Webserver started at http://127.2.218.2:33079/ using document root <none> and password file <none>
I20260923 14:39:41.044941 19599 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:41.044994 19599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:41.048128 19599 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:39:41.049947 19616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:41.051059 19599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:41.051141 19599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "391cf02b500543258d46f14cec926d29"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "953e60b06c1883677999e787ca4e45c2"
server_key_iv: "07dcaaacd38e84f175d722367edf4f2a"
server_key_version: "encryptionkey@0"
I20260923 14:39:41.051589 19599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:41.062290 19599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:41.064491 19599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:41.064651 19599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:41.065104 19599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:41.066035 19623 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 14:39:41.069276 19599 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 14:39:41.069341 19599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260923 14:39:41.069438 19599 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 14:39:41.070926 19599 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 14:39:41.070990 19599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260923 14:39:41.071190 19623 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Bootstrap starting.
I20260923 14:39:41.081063 19599 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:43593
I20260923 14:39:41.081140 19730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:43593 every 8 connection(s)
I20260923 14:39:41.081920 19599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:39:41.085026 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:41.083614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:38915)
0923 14:39:41.084081 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:41.084223 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":104,"threads_started":1}
W20260923 14:39:41.085372 19731 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:33485 (0 consecutive failures): Network error: Failed to ping master at 127.2.218.62:33485: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
I20260923 14:39:41.085819 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 19599
I20260923 14:39:41.085971 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:41.085754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:54793)
0923 14:39:41.085870 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:41.085917 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 14:39:41.086282 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:32801
--local_ip_for_outbound_sockets=127.2.218.3
--tserver_master_addrs=127.2.218.62:33485
--webserver_port=37239
--webserver_interface=127.2.218.3
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:39:41.086623 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:41.086431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:51303)
0923 14:39:41.086544 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:41.086580 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 14:39:41.086798 19731 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 14:39:41.087437 19623 log.cc:824] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:41.094619 19623 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:39:41.094992 19623 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Bootstrap complete.
I20260923 14:39:41.096328 19623 ts_tablet_manager.cc:1397] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.008s
I20260923 14:39:41.099682 19623 raft_consensus.cc:348] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:41.100121 19623 raft_consensus.cc:729] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391cf02b500543258d46f14cec926d29, State: Initialized, Role: FOLLOWER
I20260923 14:39:41.100528 19623 consensus_queue.cc:260] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:41.101370 19623 ts_tablet_manager.cc:1428] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260923 14:39:41.101533 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:41.101298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:45131)
0923 14:39:41.101417 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:41.101473 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 14:39:41.102249 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:41.102016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:48857)
0923 14:39:41.102153 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:41.102204 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 14:39:41.183387 19734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:41.183573 19734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:41.183607 19734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:41.187199 19734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:41.187240 19734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:41.187330 19734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:39:41.191788 19734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:32801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=37239
--tserver_master_addrs=127.2.218.62:33485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:41.192548 19734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:41.193322 19734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:41.201822 19742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:41.201892 19734 server_base.cc:1034] running on GCE node
W20260923 14:39:41.201880 19741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:41.201826 19744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:41.202435 19734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:41.202950 19734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:41.204147 19734 hybrid_clock.cc:648] HybridClock initialized: now 1790174381204125 us; error 46 us; skew 500 ppm
Sep 23 14:39:41 dist-test-slave-zl47 krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:41.206486 19734 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:39:41.207147 19734 webserver.cc:460] Webserver started at http://127.2.218.3:37239/ using document root <none> and password file <none>
I20260923 14:39:41.207585 19734 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:41.207636 19734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:41.210651 19734 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:39:41.212368 19751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:41.213382 19734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:41.213457 19734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "4437d75272114d0e85dec0cfcf851f3f"
format_stamp: "Formatted at 2026-09-23 14:38:59 on dist-test-slave-zl47"
server_key: "3a30f4ae3cbb577c8967184b5a7356c4"
server_key_iv: "8d66a243f6bf96191aaa9678fb335ca6"
server_key_version: "encryptionkey@0"
I20260923 14:39:41.213891 19734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:41.225960 19734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:41.228578 19734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:41.228746 19734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:41.229204 19734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:41.229846 19734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:39:41.229916 19734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:41.229970 19734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:39:41.230015 19734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:41.238654 19734 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:32801
I20260923 14:39:41.238686 19864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:32801 every 8 connection(s)
I20260923 14:39:41.239423 19734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:39:41.242491 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:41.241204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:57201)
0923 14:39:41.241780 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:41.241873 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":102,"threads_started":1}
W20260923 14:39:41.242874 19865 heartbeater.cc:644] Failed to heartbeat to 127.2.218.62:33485 (0 consecutive failures): Network error: Failed to ping master at 127.2.218.62:33485: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
I20260923 14:39:41.243402 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:41.243185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:46031)
0923 14:39:41.243289 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:41.243332 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 14:39:41.243698 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 19734
I20260923 14:39:41.244081 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:41.243902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:59695)
0923 14:39:41.244002 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:41.244039 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 14:39:41.244240 19865 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 14:39:41.961441 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:41.961021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:35695)
0923 14:39:41.961275 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:41.961325 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
I20260923 14:39:42.103183 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:42.102851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:48349)
0923 14:39:42.103067 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:42.103110 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260923 14:39:42.224133 19871 raft_consensus.cc:482] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:39:42.224299 19871 raft_consensus.cc:504] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:42.225296 19871 leader_election.cc:288] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801), 391cf02b500543258d46f14cec926d29 (127.2.218.2:43593)
Sep 23 14:39:42 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174380, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.3@KRBTEST.COM
Sep 23 14:39:42 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174380, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.2@KRBTEST.COM
I20260923 14:39:42.242051 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:42.225483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:43593 (local address 127.2.218.1:53121)
0923 14:39:42.225637 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:42.225649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:42.225744 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:42.227307 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:42.227312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:42.227748 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:42.228636 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:42.228657 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:42.230572 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:42.230576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:42.232480 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:42.232492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:42.233459 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:42.233464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:42.233798 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:42.234644 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:39:42.234680 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:42.237304 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:42.239533 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:42.239536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:42.239538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:42.239852 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:42.240216 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:42.240219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:42.240221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:42.240388 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:42.241140 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:42.241147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:42.241382 (+ 235us) client_negotiation.cc:769] Sending connection context
0923 14:39:42.241540 (+ 158us) client_negotiation.cc:240] Negotiation successful
0923 14:39:42.241858 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":505,"spinlock_wait_cycles":41216}
I20260923 14:39:42.242646 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:42.225617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:53121 (local address 127.2.218.2:43593)
0923 14:39:42.225999 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:42.226013 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:42.226057 (+ 44us) server_negotiation.cc:407] Connection header received
0923 14:39:42.226486 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:42.226491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:42.226919 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:42.227263 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:42.228851 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:42.230402 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:42.232617 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:42.233310 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:42.233481 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:42.237525 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:42.237533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:42.237549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:42.237583 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:42.239433 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:42.239964 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:42.239968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:42.239974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:42.240091 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:42.240511 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:42.240517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:42.240521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:42.240972 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:42.241117 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:42.241823 (+ 706us) server_negotiation.cc:299] Negotiation successful
0923 14:39:42.242300 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":57,"threads_started":1}
I20260923 14:39:42.242688 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:42.225491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:32801 (local address 127.2.218.1:41721)
0923 14:39:42.226153 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:42.226165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:42.226258 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:42.227710 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:42.227715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:42.227880 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:42.228636 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:42.228657 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:42.230572 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:42.230576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:42.232480 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:42.232492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:42.233459 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:42.233464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:42.233794 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:42.234644 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:39:42.234674 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:42.236777 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:42.239453 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:42.239462 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:42.239477 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:42.239778 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:42.240211 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:42.240213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:42.240215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:42.240359 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:42.241188 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:42.241190 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:42.241372 (+ 182us) client_negotiation.cc:769] Sending connection context
0923 14:39:42.241548 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 14:39:42.242515 (+ 967us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":1641,"spinlock_wait_cycles":5376,"thread_start_us":63,"threads_started":1}
I20260923 14:39:42.243172 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:42.225587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:41721 (local address 127.2.218.3:32801)
0923 14:39:42.225950 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:42.225961 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:42.226293 (+ 332us) server_negotiation.cc:407] Connection header received
0923 14:39:42.227044 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:42.227052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:42.227455 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:42.227790 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:42.228858 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:42.230400 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:42.232617 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:42.233310 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:42.233481 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:42.236999 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:42.237006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:42.237021 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:42.237056 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:42.239313 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:42.239921 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:42.239928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:42.239934 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:42.240084 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:42.240465 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:42.240471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:42.240476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:42.240972 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:42.241117 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:42.241686 (+ 569us) server_negotiation.cc:299] Negotiation successful
0923 14:39:42.241957 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":87,"threads_started":1}
I20260923 14:39:42.243957 19685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2334dddcae1456e8b40c214fec4005f" candidate_uuid: "185023dcb9a1476a856c03e7b7150b4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "391cf02b500543258d46f14cec926d29" is_pre_election: true
I20260923 14:39:42.244264 19685 raft_consensus.cc:2454] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 185023dcb9a1476a856c03e7b7150b4e in term 1.
I20260923 14:39:42.245144 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:42.244725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:40539)
0923 14:39:42.244958 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:42.245044 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260923 14:39:42.245373 19484 leader_election.cc:302] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185023dcb9a1476a856c03e7b7150b4e, 391cf02b500543258d46f14cec926d29; no voters:
W20260923 14:39:42.245636 19485 leader_election.cc:341] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801): Not found: Tablet not found: b2334dddcae1456e8b40c214fec4005f
I20260923 14:39:42.245700 19871 raft_consensus.cc:2781] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 14:39:42.245766 19871 raft_consensus.cc:482] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:39:42.245812 19871 raft_consensus.cc:3037] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 1 FOLLOWER]: Advancing to term 2
I20260923 14:39:42.247860 19871 raft_consensus.cc:504] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:42.248203 19871 leader_election.cc:288] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 2 election: Requested vote from peers 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801), 391cf02b500543258d46f14cec926d29 (127.2.218.2:43593)
I20260923 14:39:42.248560 19685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2334dddcae1456e8b40c214fec4005f" candidate_uuid: "185023dcb9a1476a856c03e7b7150b4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "391cf02b500543258d46f14cec926d29"
I20260923 14:39:42.248693 19685 raft_consensus.cc:3037] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 1 FOLLOWER]: Advancing to term 2
W20260923 14:39:42.248751 19485 leader_election.cc:341] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801): Not found: Tablet not found: b2334dddcae1456e8b40c214fec4005f
I20260923 14:39:42.250486 19685 raft_consensus.cc:2454] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185023dcb9a1476a856c03e7b7150b4e in term 2.
I20260923 14:39:42.250798 19484 leader_election.cc:302] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 185023dcb9a1476a856c03e7b7150b4e, 391cf02b500543258d46f14cec926d29; no voters: 4437d75272114d0e85dec0cfcf851f3f
I20260923 14:39:42.251008 19871 raft_consensus.cc:2781] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 2 FOLLOWER]: Leader election won for term 2
I20260923 14:39:42.251211 19871 raft_consensus.cc:686] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [term 2 LEADER]: Becoming Leader. State: Replica: 185023dcb9a1476a856c03e7b7150b4e, State: Running, Role: LEADER
I20260923 14:39:42.251575 19871 consensus_queue.cc:237] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:42.251816 19874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:42.251601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:41913)
0923 14:39:42.251730 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:42.251776 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 14:39:42.817445 19685 raft_consensus.cc:1261] T b2334dddcae1456e8b40c214fec4005f P 391cf02b500543258d46f14cec926d29 [term 2 FOLLOWER]: Refusing update from remote peer 185023dcb9a1476a856c03e7b7150b4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260923 14:39:42.818168 19878 consensus_queue.cc:1035] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260923 14:39:42.819900 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:42.819599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:49533)
0923 14:39:42.819787 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:42.819823 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":42,"threads_started":1}
W20260923 14:39:42.822332 19485 consensus_peers.cc:487] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e -> Peer 4437d75272114d0e85dec0cfcf851f3f (127.2.218.3:32801): Couldn't send request to peer 4437d75272114d0e85dec0cfcf851f3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2334dddcae1456e8b40c214fec4005f. This is attempt 1: this message will repeat every 5th retry.
I20260923 14:39:42.822510 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:42.821867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:41943)
0923 14:39:42.822256 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:42.822325 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":89,"threads_started":1}
I20260923 14:39:42.822643 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:42.822376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:59847)
0923 14:39:42.822532 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:42.822571 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 14:39:43.246148 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:43.245794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:40703)
0923 14:39:43.245989 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:43.246032 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
I20260923 14:39:43.256700 19888 ts_tablet_manager.cc:927] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Initiating tablet copy from peer 185023dcb9a1476a856c03e7b7150b4e (127.2.218.1:38711)
I20260923 14:39:43.257200 19888 tablet_copy_client.cc:316] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: tablet copy: Beginning tablet copy session from remote peer at address 127.2.218.1:38711
Sep 23 14:39:43 dist-test-slave-zl47 krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174381, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.1@KRBTEST.COM
I20260923 14:39:43.266608 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:43.257583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:38711 (local address 127.2.218.3:36345)
0923 14:39:43.257688 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:43.257710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:43.257822 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:43.258506 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:43.258510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:43.258845 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:43.258955 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:43.258959 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:43.259832 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:43.259834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:43.260391 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:43.260398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:43.260912 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:43.260914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:43.260981 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:43.261548 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:39:43.261559 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:43.263029 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:43.264910 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:43.264919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:43.264925 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:43.265146 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:43.265458 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:43.265461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:43.265462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:43.265548 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:43.265994 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:43.265999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:43.266142 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 14:39:43.266293 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 14:39:43.266462 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 14:39:43.266660 19890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:43.257720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:36345 (local address 127.2.218.1:38711)
0923 14:39:43.258050 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:43.258059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:43.258079 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:39:43.258179 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:43.258184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:43.258364 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:43.258472 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:43.259059 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:43.259710 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:43.260488 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:43.260803 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:43.260900 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:43.263175 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:43.263184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:43.263193 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:43.263214 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:43.264749 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:43.265261 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:43.265266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:43.265268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:43.265359 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:43.265637 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:43.265642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:43.265643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:43.265878 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:43.266018 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:43.266385 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 14:39:43.266514 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":68,"threads_started":1}
I20260923 14:39:43.267560 19570 tablet_copy_service.cc:140] P 185023dcb9a1476a856c03e7b7150b4e: Received BeginTabletCopySession request for tablet b2334dddcae1456e8b40c214fec4005f from peer 4437d75272114d0e85dec0cfcf851f3f ({username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:36345)
I20260923 14:39:43.267719 19570 tablet_copy_service.cc:161] P 185023dcb9a1476a856c03e7b7150b4e: Beginning new tablet copy session on tablet b2334dddcae1456e8b40c214fec4005f from peer 4437d75272114d0e85dec0cfcf851f3f at {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:36345: session id = 4437d75272114d0e85dec0cfcf851f3f-b2334dddcae1456e8b40c214fec4005f
I20260923 14:39:43.268710 19570 tablet_copy_source_session.cc:205] T b2334dddcae1456e8b40c214fec4005f P 185023dcb9a1476a856c03e7b7150b4e: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 14:39:43.272042 19888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2334dddcae1456e8b40c214fec4005f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:43.277591 19888 tablet_copy_client.cc:791] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: tablet copy: Starting download of 0 data blocks...
I20260923 14:39:43.277784 19888 tablet_copy_client.cc:655] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: tablet copy: Starting download of 1 WAL segments...
I20260923 14:39:43.279227 19888 tablet_copy_client.cc:531] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 14:39:43.280577 19888 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Bootstrap starting.
I20260923 14:39:43.285341 19888 log.cc:824] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:43.290560 19888 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 14:39:43.290824 19888 tablet_bootstrap.cc:492] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Bootstrap complete.
I20260923 14:39:43.291092 19888 ts_tablet_manager.cc:1397] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.000s
I20260923 14:39:43.292922 19888 raft_consensus.cc:348] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:43.293144 19888 raft_consensus.cc:729] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4437d75272114d0e85dec0cfcf851f3f, State: Initialized, Role: FOLLOWER
I20260923 14:39:43.293514 19888 consensus_queue.cc:260] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4437d75272114d0e85dec0cfcf851f3f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 32801 } } peers { permanent_uuid: "391cf02b500543258d46f14cec926d29" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 43593 } } peers { permanent_uuid: "185023dcb9a1476a856c03e7b7150b4e" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 38711 } }
I20260923 14:39:43.294386 19888 ts_tablet_manager.cc:1428] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 14:39:43.294492 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:43.294273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:55833)
0923 14:39:43.294380 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:43.294438 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 14:39:43.294893 19570 tablet_copy_service.cc:342] P 185023dcb9a1476a856c03e7b7150b4e: Request end of tablet copy session 4437d75272114d0e85dec0cfcf851f3f-b2334dddcae1456e8b40c214fec4005f received from {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:36345
I20260923 14:39:43.294981 19570 tablet_copy_service.cc:433] P 185023dcb9a1476a856c03e7b7150b4e: ending tablet copy session 4437d75272114d0e85dec0cfcf851f3f-b2334dddcae1456e8b40c214fec4005f on tablet b2334dddcae1456e8b40c214fec4005f with peer 4437d75272114d0e85dec0cfcf851f3f
I20260923 14:39:43.295130 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:43.294918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:40223)
0923 14:39:43.295032 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:43.295078 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 14:39:43.820091 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:43.819784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:53715)
0923 14:39:43.819973 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:43.820011 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":39,"threads_started":1}
I20260923 14:39:43.820122 19819 raft_consensus.cc:1203] T b2334dddcae1456e8b40c214fec4005f P 4437d75272114d0e85dec0cfcf851f3f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260923 14:39:43.821581 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:43.821264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:59815)
0923 14:39:43.821443 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:43.821487 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":58,"threads_started":1}
I20260923 14:39:43.823578 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:43.823272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:53093)
0923 14:39:43.823466 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:43.823508 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
I20260923 14:39:44.821164 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:44.820779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:55961)
0923 14:39:44.821000 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:44.821049 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260923 14:39:44.822532 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:44.822168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:43679)
0923 14:39:44.822414 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:44.822465 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260923 14:39:44.824483 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:44.824155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:54467)
0923 14:39:44.824371 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:44.824413 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":54,"threads_started":1}
I20260923 14:39:45.822360 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:45.821892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:49431)
0923 14:39:45.822197 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:45.822249 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260923 14:39:45.823490 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:45.823075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:59389)
0923 14:39:45.823304 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:45.823388 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260923 14:39:45.825412 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:45.825045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:49473)
0923 14:39:45.825248 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:45.825315 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260923 14:39:46.823549 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:46.823140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:49259)
0923 14:39:46.823399 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:46.823456 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
I20260923 14:39:46.824292 19905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:46.824048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:48037)
0923 14:39:46.824171 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:46.824229 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 14:39:46.824688 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:46.824298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:49885)
0923 14:39:46.824576 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:46.824619 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":62,"threads_started":1}
I20260923 14:39:46.826448 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:46.826050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:41595)
0923 14:39:46.826319 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:46.826379 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
I20260923 14:39:47.825255 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:47.824884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:52281)
0923 14:39:47.825126 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:47.825172 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":94,"threads_started":1}
I20260923 14:39:47.825678 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:47.825260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:46765)
0923 14:39:47.825543 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:47.825607 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":77,"threads_started":1}
I20260923 14:39:47.827306 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:47.826994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:60221)
0923 14:39:47.827190 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:47.827236 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260923 14:39:48.826496 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:48.825974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:51741)
0923 14:39:48.826311 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:48.826372 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":115,"threads_started":1}
I20260923 14:39:48.826809 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:48.826388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:52367)
0923 14:39:48.826637 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:48.826699 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260923 14:39:48.828218 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:48.827917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:39275)
0923 14:39:48.828104 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:48.828148 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":55,"threads_started":1}
I20260923 14:39:49.827471 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:49.827104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:43325)
0923 14:39:49.827332 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:49.827385 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260923 14:39:49.827893 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:49.827464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:46483)
0923 14:39:49.827742 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:49.827803 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":59,"threads_started":1}
I20260923 14:39:49.829252 19917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:49.828827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:55721)
0923 14:39:49.829073 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:49.829140 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":58,"threads_started":1}
I20260923 14:39:50.828591 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:50.828232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:59663)
0923 14:39:50.828460 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:50.828506 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260923 14:39:50.828823 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:50.828473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:42683)
0923 14:39:50.828665 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:50.828732 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260923 14:39:50.830181 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:50.829834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:55751)
0923 14:39:50.830021 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:50.830063 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
I20260923 14:39:51.829870 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:51.829322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:56555)
0923 14:39:51.829696 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:51.829741 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":72,"threads_started":1}
I20260923 14:39:51.829908 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:51.829543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:42855)
0923 14:39:51.829758 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:51.829820 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20260923 14:39:51.831121 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:51.830789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:38347)
0923 14:39:51.830998 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:51.831047 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260923 14:39:52.830844 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:52.830437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:51661)
0923 14:39:52.830665 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:52.830727 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260923 14:39:52.831053 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:52.830656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:43489)
0923 14:39:52.830860 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:52.830946 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260923 14:39:52.832026 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:52.831679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:40783)
0923 14:39:52.831922 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:52.831959 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":62,"threads_started":1}
I20260923 14:39:53.832038 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:53.831537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:52665)
0923 14:39:53.831845 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:53.831915 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":105,"threads_started":1}
I20260923 14:39:53.832213 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:53.831795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:33777)
0923 14:39:53.832037 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:53.832087 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":91,"threads_started":1}
I20260923 14:39:53.832994 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:53.832645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:43201)
0923 14:39:53.832858 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:53.832919 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":59,"threads_started":1}
I20260923 14:39:54.833005 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:54.832624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:42263)
0923 14:39:54.832868 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:54.832922 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260923 14:39:54.833422 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:54.832994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:35783)
0923 14:39:54.833291 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:54.833337 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":72,"threads_started":1}
I20260923 14:39:54.833937 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:54.833606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:59403)
0923 14:39:54.833811 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:54.833864 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260923 14:39:55.834407 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:55.833691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.1:43449)
0923 14:39:55.833958 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:55.834037 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":63,"threads_started":1}
I20260923 14:39:55.834488 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:55.834070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.3:50813)
0923 14:39:55.834320 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:55.834373 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260923 14:39:55.834946 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:55.834569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33485 (local address 127.2.218.2:38833)
0923 14:39:55.834835 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:55.834878 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.62:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260923 14:39:56.253132 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 19425
I20260923 14:39:56.259281 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 19599
I20260923 14:39:56.264715 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 19734
2026-09-23T14:39:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58585 ms)
[----------] 30 tests from SecurityITest (222927 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:39:56 dist-test-slave-zl47 krb5kdc[19938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:39:56 dist-test-slave-zl47 krb5kdc[19938](info): set up 2 sockets
Sep 23 14:39:56 dist-test-slave-zl47 krb5kdc[19938](info): commencing operation
krb5kdc: starting...
W20260923 14:39:57.238341 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:39:57 dist-test-slave-zl47 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:39:57.344337 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33281
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:33281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:39:57.440053 19951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:57.440237 19951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:57.440268 19951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:57.443572 19951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:39:57.443614 19951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:57.443631 19951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:57.443645 19951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:39:57.443657 19951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:39:57.448252 19951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:33281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:57.449009 19951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:57.449826 19951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:57.457414 19956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:57.457492 19957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:57.457540 19959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:57.457434 19951 server_base.cc:1034] running on GCE node
I20260923 14:39:57.458252 19951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:57.459161 19951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:57.460355 19951 hybrid_clock.cc:648] HybridClock initialized: now 1790174397460351 us; error 41 us; skew 500 ppm
Sep 23 14:39:57 dist-test-slave-zl47 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174397, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:57.463020 19951 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:39:57.463763 19951 webserver.cc:460] Webserver started at http://127.2.218.62:32993/ using document root <none> and password file <none>
I20260923 14:39:57.464263 19951 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:57.464311 19951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:57.464455 19951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:39:57.466264 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "5591621200e14538aa677998bbca1582"
format_stamp: "Formatted at 2026-09-23 14:39:57 on dist-test-slave-zl47"
server_key: "3e1eca5e266d7f90ed069109f88f3d02"
server_key_iv: "7abcdfa791ec1dce3d767fde3bed7c53"
server_key_version: "encryptionkey@0"
I20260923 14:39:57.466698 19951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "5591621200e14538aa677998bbca1582"
format_stamp: "Formatted at 2026-09-23 14:39:57 on dist-test-slave-zl47"
server_key: "3e1eca5e266d7f90ed069109f88f3d02"
server_key_iv: "7abcdfa791ec1dce3d767fde3bed7c53"
server_key_version: "encryptionkey@0"
I20260923 14:39:57.469821 19951 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:39:57.472110 19966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:57.473345 19951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:57.473459 19951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "5591621200e14538aa677998bbca1582"
format_stamp: "Formatted at 2026-09-23 14:39:57 on dist-test-slave-zl47"
server_key: "3e1eca5e266d7f90ed069109f88f3d02"
server_key_iv: "7abcdfa791ec1dce3d767fde3bed7c53"
server_key_version: "encryptionkey@0"
I20260923 14:39:57.473587 19951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:57.487818 19951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:57.490499 19951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:57.490690 19951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:57.497066 19951 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:33281
I20260923 14:39:57.497139 20018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:33281 every 8 connection(s)
I20260923 14:39:57.497949 19951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:39:57.498860 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 19951
I20260923 14:39:57.498958 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:39:57.499249 2920 external_mini_cluster.cc:1428] Setting key 1434e0740c4755bac72cbb23d2a51728
I20260923 14:39:57.500890 20019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:57.505507 20019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582: Bootstrap starting.
Sep 23 14:39:57 dist-test-slave-zl47 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:39:57.507742 20019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:57.508462 20019 log.cc:824] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:57.510192 20019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582: No bootstrap required, opened a new log
I20260923 14:39:57.512171 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:57.500583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.2.218.62:33281)
0923 14:39:57.501058 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:57.501074 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:57.501129 (+ 55us) server_negotiation.cc:407] Connection header received
0923 14:39:57.501694 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:57.501709 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:57.501992 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:57.502334 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:57.503099 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:57.503968 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:57.504809 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:57.505256 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:57.505422 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:57.507957 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:57.507962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:57.507972 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:57.507996 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:57.509610 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:57.510038 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:57.510049 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:57.510052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:57.510260 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:57.510535 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:57.510540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:57.510541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:57.510885 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:57.511079 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:57.511333 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 14:39:57.511588 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":130,"threads_started":1}
I20260923 14:39:57.513455 20019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5591621200e14538aa677998bbca1582" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33281 } }
I20260923 14:39:57.513631 20019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:57.513660 20019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5591621200e14538aa677998bbca1582, State: Initialized, Role: FOLLOWER
I20260923 14:39:57.514098 20019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5591621200e14538aa677998bbca1582" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33281 } }
I20260923 14:39:57.514248 20019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:39:57.514312 20019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:39:57.514406 20019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:57.515118 20019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5591621200e14538aa677998bbca1582" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33281 } }
I20260923 14:39:57.515440 20019 leader_election.cc:302] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5591621200e14538aa677998bbca1582; no voters:
I20260923 14:39:57.515718 20019 leader_election.cc:288] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:39:57.515828 20024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:39:57.516057 20024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [term 1 LEADER]: Becoming Leader. State: Replica: 5591621200e14538aa677998bbca1582, State: Running, Role: LEADER
I20260923 14:39:57.516359 20024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5591621200e14538aa677998bbca1582" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33281 } }
I20260923 14:39:57.516736 20019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:39:57.517719 20026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5591621200e14538aa677998bbca1582. Latest consensus state: current_term: 1 leader_uuid: "5591621200e14538aa677998bbca1582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5591621200e14538aa677998bbca1582" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33281 } } }
I20260923 14:39:57.517855 20026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [sys.catalog]: This master's current role is: LEADER
I20260923 14:39:57.518028 20025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5591621200e14538aa677998bbca1582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5591621200e14538aa677998bbca1582" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 33281 } } }
I20260923 14:39:57.518401 20025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582 [sys.catalog]: This master's current role is: LEADER
W20260923 14:39:57.521108 20039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:39:57.521186 20039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:39:57.521262 20036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:39:57.522089 20036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:39:57.526767 20036 catalog_manager.cc:1347] Generated new cluster ID: a160e964739a494d8e9752ada593837b
I20260923 14:39:57.526882 20036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:39:57.534420 20036 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:39:57.535455 20036 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:39:57.539913 20036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582: Generated new TSK 0
I20260923 14:39:57.540588 20036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:39:57.565515 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33281
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:39:57.658676 20045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:57.658900 20045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:57.658968 20045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:57.662256 20045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:57.662322 20045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:57.662398 20045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:39:57.666679 20045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:57.667459 20045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:57.668304 20045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:57.676378 20051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:57.676403 20053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:57.676486 20045 server_base.cc:1034] running on GCE node
W20260923 14:39:57.676401 20050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:57.676957 20045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:57.677443 20045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:57.678617 20045 hybrid_clock.cc:648] HybridClock initialized: now 1790174397678594 us; error 37 us; skew 500 ppm
Sep 23 14:39:57 dist-test-slave-zl47 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174397, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:57.680976 20045 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:39:57.681715 20045 webserver.cc:460] Webserver started at http://127.2.218.1:40469/ using document root <none> and password file <none>
I20260923 14:39:57.682303 20045 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:57.682399 20045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:57.682622 20045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:39:57.684410 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "ab9d60a7c4394910a8bd0ba3697ddefe"
format_stamp: "Formatted at 2026-09-23 14:39:57 on dist-test-slave-zl47"
server_key: "db5f684e2477fb72e86608ea2f539b00"
server_key_iv: "5717b80c715a1f1d046342c22d6bf8ae"
server_key_version: "encryptionkey@0"
I20260923 14:39:57.684938 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "ab9d60a7c4394910a8bd0ba3697ddefe"
format_stamp: "Formatted at 2026-09-23 14:39:57 on dist-test-slave-zl47"
server_key: "db5f684e2477fb72e86608ea2f539b00"
server_key_iv: "5717b80c715a1f1d046342c22d6bf8ae"
server_key_version: "encryptionkey@0"
I20260923 14:39:57.688202 20045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:39:57.690351 20060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:57.691426 20045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:57.691560 20045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "ab9d60a7c4394910a8bd0ba3697ddefe"
format_stamp: "Formatted at 2026-09-23 14:39:57 on dist-test-slave-zl47"
server_key: "db5f684e2477fb72e86608ea2f539b00"
server_key_iv: "5717b80c715a1f1d046342c22d6bf8ae"
server_key_version: "encryptionkey@0"
I20260923 14:39:57.691660 20045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:57.703332 20045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:57.705825 20045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:57.706022 20045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:57.706544 20045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:57.707273 20045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:39:57.707342 20045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:57.707409 20045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:39:57.707456 20045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:57.715607 20045 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:36607
I20260923 14:39:57.715624 20173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:36607 every 8 connection(s)
I20260923 14:39:57.716358 20045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:39:57.720594 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 20045
I20260923 14:39:57.720727 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:39:57.721016 2920 external_mini_cluster.cc:1428] Setting key f17542640e5dd158c24c22c00579b12a
Sep 23 14:39:57 dist-test-slave-zl47 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174397, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:39:57.729244 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:57.717833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:51713 (local address 127.2.218.62:33281)
0923 14:39:57.717995 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:57.717999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:57.718739 (+ 740us) server_negotiation.cc:407] Connection header received
0923 14:39:57.719815 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:57.719818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:57.719859 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:57.719928 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:57.721143 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:57.721692 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:57.723092 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:57.723285 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:57.723352 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:57.725829 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:57.725835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:57.725840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:57.725867 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:57.727443 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:57.727889 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:57.727895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:57.727899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:57.727986 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:57.728277 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:57.728281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:57.728285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:57.728472 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:57.728570 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:57.728958 (+ 388us) server_negotiation.cc:299] Negotiation successful
0923 14:39:57.729078 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:39:57.729733 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:57.718194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33281 (local address 127.2.218.1:51713)
0923 14:39:57.718570 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:57.718625 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:57.719647 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:57.720063 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:57.720069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:57.720414 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:57.720972 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:57.720990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:57.721789 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:57.721793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:57.722980 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:57.722993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:57.723422 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:57.723428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:57.723632 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:57.724264 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:39:57.724280 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:57.725683 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:57.727551 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:57.727554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:57.727563 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:57.727783 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:57.728068 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:57.728070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:57.728071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:57.728184 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:57.728570 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:57.728573 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:57.728746 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 14:39:57.728878 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 14:39:57.729069 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
I20260923 14:39:57.730803 20174 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33281
I20260923 14:39:57.730963 20174 heartbeater.cc:464] Registering TS with master...
I20260923 14:39:57.731343 20174 heartbeater.cc:511] Master 127.2.218.62:33281 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
I20260923 14:39:57.732625 19983 ts_manager.cc:194] Registered new tserver with Master: ab9d60a7c4394910a8bd0ba3697ddefe (127.2.218.1:36607)
I20260923 14:39:57.733675 19983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:51713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:39:57.746528 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33281
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:39:57.839632 20179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:57.839819 20179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:57.839882 20179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:57.843211 20179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:57.843256 20179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:57.843362 20179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:39:57.847847 20179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:57.848588 20179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:57.849395 20179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:57.857331 20187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:57.857335 20185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:57.857479 20179 server_base.cc:1034] running on GCE node
W20260923 14:39:57.857354 20184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:57.857884 20179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:57.858501 20179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:57.859675 20179 hybrid_clock.cc:648] HybridClock initialized: now 1790174397859660 us; error 28 us; skew 500 ppm
Sep 23 14:39:57 dist-test-slave-zl47 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174397, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:57.862012 20179 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:39:57.862717 20179 webserver.cc:460] Webserver started at http://127.2.218.2:42405/ using document root <none> and password file <none>
I20260923 14:39:57.863186 20179 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:57.863263 20179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:57.863456 20179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:39:57.865084 20179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "a9788b4256094d8a8085ed0443e0075d"
format_stamp: "Formatted at 2026-09-23 14:39:57 on dist-test-slave-zl47"
server_key: "d6a29817b69410b53dcd2c7792ef9b89"
server_key_iv: "f3ab53bff2fa207f461f20f828c4cc1b"
server_key_version: "encryptionkey@0"
I20260923 14:39:57.865537 20179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "a9788b4256094d8a8085ed0443e0075d"
format_stamp: "Formatted at 2026-09-23 14:39:57 on dist-test-slave-zl47"
server_key: "d6a29817b69410b53dcd2c7792ef9b89"
server_key_iv: "f3ab53bff2fa207f461f20f828c4cc1b"
server_key_version: "encryptionkey@0"
I20260923 14:39:57.868696 20179 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:39:57.870781 20194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:57.871711 20179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:57.871831 20179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "a9788b4256094d8a8085ed0443e0075d"
format_stamp: "Formatted at 2026-09-23 14:39:57 on dist-test-slave-zl47"
server_key: "d6a29817b69410b53dcd2c7792ef9b89"
server_key_iv: "f3ab53bff2fa207f461f20f828c4cc1b"
server_key_version: "encryptionkey@0"
I20260923 14:39:57.871928 20179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:57.884975 20179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:57.887622 20179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:57.887840 20179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:57.888314 20179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:57.888967 20179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:39:57.889029 20179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:57.889086 20179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:39:57.889139 20179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:57.896834 20179 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:40415
I20260923 14:39:57.896863 20307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:40415 every 8 connection(s)
I20260923 14:39:57.897500 20179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:39:57.901136 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 20179
I20260923 14:39:57.901252 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:39:57.901496 2920 external_mini_cluster.cc:1428] Setting key fc88b23d9cbe3a9f17e7065db8c5b1a3
Sep 23 14:39:57 dist-test-slave-zl47 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174397, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:39:57.909317 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:57.898912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:35751 (local address 127.2.218.62:33281)
0923 14:39:57.899028 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:57.899031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:57.899661 (+ 630us) server_negotiation.cc:407] Connection header received
0923 14:39:57.900648 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:57.900650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:57.900703 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:57.900794 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:57.901883 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:57.902531 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:57.903604 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:57.903825 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:57.903937 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:57.905966 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:57.905972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:57.905978 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:57.906009 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:57.907425 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:57.907859 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:57.907864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:57.907878 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:57.907959 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:57.908289 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:57.908294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:57.908296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:57.908555 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:57.908657 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:57.909058 (+ 401us) server_negotiation.cc:299] Negotiation successful
0923 14:39:57.909172 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:39:57.909897 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:57.899169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33281 (local address 127.2.218.2:35751)
0923 14:39:57.899532 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:57.899566 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:57.900483 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:57.900931 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:57.900936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:57.901276 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:57.901733 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:57.901745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:57.902657 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:57.902660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:57.903481 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:57.903490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:57.903966 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:57.903970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:57.904119 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:57.904631 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:39:57.904644 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:57.905818 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:57.907533 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:57.907538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:57.907549 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:57.907763 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:57.908056 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:57.908059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:57.908060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:57.908201 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:57.908655 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:57.908660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:57.908856 (+ 196us) client_negotiation.cc:769] Sending connection context
0923 14:39:57.909016 (+ 160us) client_negotiation.cc:240] Negotiation successful
0923 14:39:57.909267 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
I20260923 14:39:57.910813 20308 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33281
I20260923 14:39:57.910966 20308 heartbeater.cc:464] Registering TS with master...
I20260923 14:39:57.911327 20308 heartbeater.cc:511] Master 127.2.218.62:33281 requested a full tablet report, sending...
I20260923 14:39:57.912150 19983 ts_manager.cc:194] Registered new tserver with Master: a9788b4256094d8a8085ed0443e0075d (127.2.218.2:40415)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
I20260923 14:39:57.912787 19983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:35751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:39:57.926859 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33281
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:39:58.020287 20313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:58.020493 20313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:58.020529 20313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:58.023823 20313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:58.023872 20313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:58.023921 20313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:39:58.028070 20313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:58.028782 20313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:58.029510 20313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:58.037642 20319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:58.037649 20318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:58.037756 20321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:58.038285 20313 server_base.cc:1034] running on GCE node
I20260923 14:39:58.038683 20313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:58.039149 20313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:58.040328 20313 hybrid_clock.cc:648] HybridClock initialized: now 1790174398040274 us; error 66 us; skew 500 ppm
Sep 23 14:39:58 dist-test-slave-zl47 krb5kdc[19938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:39:58.043496 20313 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:39:58.044262 20313 webserver.cc:460] Webserver started at http://127.2.218.3:37615/ using document root <none> and password file <none>
I20260923 14:39:58.044726 20313 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:58.044782 20313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:58.044994 20313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:39:58.046690 20313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "56f1205eb08342ec92bfaf4c04b1fa42"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "854f2a7d8f303adc239bd21d67e54279"
server_key_iv: "e2b1a49ee64846bed43a7094a0ce11b7"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.047184 20313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "56f1205eb08342ec92bfaf4c04b1fa42"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "854f2a7d8f303adc239bd21d67e54279"
server_key_iv: "e2b1a49ee64846bed43a7094a0ce11b7"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.050238 20313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:39:58.052273 20328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:58.053252 20313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:58.053375 20313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "56f1205eb08342ec92bfaf4c04b1fa42"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "854f2a7d8f303adc239bd21d67e54279"
server_key_iv: "e2b1a49ee64846bed43a7094a0ce11b7"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.053476 20313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:58.077756 20313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:58.080107 20313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:58.080283 20313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:58.080749 20313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:58.081488 20313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:39:58.081557 20313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:58.081612 20313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:39:58.081660 20313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:58.089743 20313 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:36465
I20260923 14:39:58.089766 20441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:36465 every 8 connection(s)
I20260923 14:39:58.090499 20313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:39:58.091219 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 20313
I20260923 14:39:58.091325 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:39:58.091596 2920 external_mini_cluster.cc:1428] Setting key af650057a51a10f609b1f8374dcf6853
Sep 23 14:39:58 dist-test-slave-zl47 krb5kdc[19938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174398, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:39:58.101961 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.092097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:57399 (local address 127.2.218.62:33281)
0923 14:39:58.092226 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.092230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.093091 (+ 861us) server_negotiation.cc:407] Connection header received
0923 14:39:58.094219 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.094221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.094261 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.094319 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:58.095292 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.095815 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.096815 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.097003 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.097071 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.099184 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:58.099187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:58.099188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:58.099208 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:58.100496 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:58.100886 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:58.100888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:58.100889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:58.100950 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:58.101212 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:58.101213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:58.101214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:58.101350 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:58.101428 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.101789 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.101866 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:39:58.102626 20444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.092498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:33281 (local address 127.2.218.3:57399)
0923 14:39:58.092960 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:58.093004 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:58.093997 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:58.094527 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:58.094533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:58.094819 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:58.095181 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.095192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.095933 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.095935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.096717 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.096726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.097114 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.097117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.097241 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.097698 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:39:58.097712 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:58.099039 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:58.100581 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:58.100585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:58.100593 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:58.100804 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:58.101017 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:39:58.101019 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:39:58.101020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:39:58.101141 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:39:58.101446 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:58.101450 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:58.101620 (+ 170us) client_negotiation.cc:769] Sending connection context
0923 14:39:58.101755 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 14:39:58.101953 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":156,"threads_started":1}
I20260923 14:39:58.103595 20442 heartbeater.cc:347] Connected to a master server at 127.2.218.62:33281
I20260923 14:39:58.103806 20442 heartbeater.cc:464] Registering TS with master...
I20260923 14:39:58.104315 20442 heartbeater.cc:511] Master 127.2.218.62:33281 requested a full tablet report, sending...
I20260923 14:39:58.105206 19983 ts_manager.cc:194] Registered new tserver with Master: 56f1205eb08342ec92bfaf4c04b1fa42 (127.2.218.3:36465)
I20260923 14:39:58.105623 19983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:57399
I20260923 14:39:58.116367 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:39:58.124740 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.118145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.2.218.62:33281)
0923 14:39:58.118268 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.118273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.118455 (+ 182us) server_negotiation.cc:407] Connection header received
0923 14:39:58.118590 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.118593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.118631 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.118716 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:58.119329 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.119755 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.120521 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.120721 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.120825 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.121996 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:58.121999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:58.122002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:39:58.122027 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:58.123162 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:58.123569 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:58.123573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:58.123574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:58.123655 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:39:58.123925 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:39:58.123929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:39:58.123931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:39:58.124122 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:58.124249 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.124493 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.124592 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:39:58.126974 19983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:39:58.129511 19983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:39:58.141786 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.136668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:36465 (local address 127.0.0.1:34804)
0923 14:39:58.137146 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:58.137186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:58.137281 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:58.137913 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:58.137917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:58.137936 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:58.138041 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.138048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.139208 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.139213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.140182 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.140191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.141281 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.141285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.141397 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.141437 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 14:39:58.141550 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 14:39:58.141654 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":21,"thread_start_us":67,"threads_started":1}
I20260923 14:39:58.141798 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.136763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.2.218.3:36465)
0923 14:39:58.137517 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.137527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.137546 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:39:58.137604 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.137609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.137766 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.138199 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:58.138321 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.139098 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.140298 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.141154 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.141246 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.141353 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.141589 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.141666 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":94,"threads_started":1}
I20260923 14:39:58.141930 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.136866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:36607 (local address 127.0.0.1:48684)
0923 14:39:58.137371 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:58.137385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:58.137465 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:58.138321 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:58.138323 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:58.138336 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:58.138416 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.138421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.139678 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.139681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.140707 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.140716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.141599 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.141602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.141679 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.141701 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:39:58.141748 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 14:39:58.141804 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":33,"thread_start_us":69,"threads_started":1}
I20260923 14:39:58.142506 20456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.137251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40415 (local address 127.0.0.1:47534)
0923 14:39:58.138090 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:58.138123 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:58.138198 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:58.138581 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:58.138583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:58.138598 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:58.138672 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.138676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.139795 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.139799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.140832 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.140841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.141918 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.141921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.142015 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.142038 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 14:39:58.142324 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 14:39:58.142397 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":36,"thread_start_us":75,"threads_started":1}
I20260923 14:39:58.142544 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.137446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.2.218.1:36607)
0923 14:39:58.137883 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.137895 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.137919 (+ 24us) server_negotiation.cc:407] Connection header received
0923 14:39:58.137995 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.138000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.138190 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.138356 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:58.138516 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.139541 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.140827 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.141475 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.142180 (+ 705us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.142300 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.142345 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.142415 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":147,"threads_started":1}
I20260923 14:39:58.142758 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.137656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.2.218.2:40415)
0923 14:39:58.138078 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.138088 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.138198 (+ 110us) server_negotiation.cc:407] Connection header received
0923 14:39:58.138298 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.138305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.138460 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.138605 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:58.138752 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.139669 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.140958 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.141652 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.141741 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.141856 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.142378 (+ 522us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.142470 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":128,"threads_started":1}
I20260923 14:39:58.144533 20376 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf468ddb9be4c0fa768da1990276acd (DEFAULT_TABLE table=test-table [id=9ad7b601662649ad8b9a88babc3d65ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:39:58.144778 20108 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf468ddb9be4c0fa768da1990276acd (DEFAULT_TABLE table=test-table [id=9ad7b601662649ad8b9a88babc3d65ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:39:58.145395 20376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf468ddb9be4c0fa768da1990276acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:58.145682 20108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf468ddb9be4c0fa768da1990276acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:58.150512 20459 tablet_bootstrap.cc:492] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: Bootstrap starting.
I20260923 14:39:58.151690 20460 tablet_bootstrap.cc:492] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe: Bootstrap starting.
I20260923 14:39:58.151993 20242 tablet_service.cc:1467] Processing CreateTablet for tablet 0cf468ddb9be4c0fa768da1990276acd (DEFAULT_TABLE table=test-table [id=9ad7b601662649ad8b9a88babc3d65ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:39:58.152695 20459 tablet_bootstrap.cc:654] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:58.152918 20242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cf468ddb9be4c0fa768da1990276acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:58.153656 20459 log.cc:824] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:58.153946 20460 tablet_bootstrap.cc:654] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:58.154657 20460 log.cc:824] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:58.155432 20459 tablet_bootstrap.cc:492] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: No bootstrap required, opened a new log
I20260923 14:39:58.155613 20459 ts_tablet_manager.cc:1397] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:39:58.156342 20460 tablet_bootstrap.cc:492] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe: No bootstrap required, opened a new log
I20260923 14:39:58.156497 20460 ts_tablet_manager.cc:1397] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:39:58.158658 20463 tablet_bootstrap.cc:492] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: Bootstrap starting.
I20260923 14:39:58.159010 20459 raft_consensus.cc:348] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 } } peers { permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 } } peers { permanent_uuid: "a9788b4256094d8a8085ed0443e0075d" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40415 } }
I20260923 14:39:58.159260 20459 raft_consensus.cc:374] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:58.159322 20459 raft_consensus.cc:729] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f1205eb08342ec92bfaf4c04b1fa42, State: Initialized, Role: FOLLOWER
I20260923 14:39:58.159719 20459 consensus_queue.cc:260] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 } } peers { permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 } } peers { permanent_uuid: "a9788b4256094d8a8085ed0443e0075d" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40415 } }
I20260923 14:39:58.159723 20460 raft_consensus.cc:348] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 } } peers { permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 } } peers { permanent_uuid: "a9788b4256094d8a8085ed0443e0075d" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40415 } }
I20260923 14:39:58.159915 20460 raft_consensus.cc:374] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:58.159960 20460 raft_consensus.cc:729] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9d60a7c4394910a8bd0ba3697ddefe, State: Initialized, Role: FOLLOWER
I20260923 14:39:58.160318 20442 heartbeater.cc:503] Master 127.2.218.62:33281 was elected leader, sending a full tablet report...
I20260923 14:39:58.160328 20460 consensus_queue.cc:260] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 } } peers { permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 } } peers { permanent_uuid: "a9788b4256094d8a8085ed0443e0075d" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40415 } }
I20260923 14:39:58.160858 20463 tablet_bootstrap.cc:654] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:58.161129 20460 ts_tablet_manager.cc:1428] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:39:58.161286 20174 heartbeater.cc:503] Master 127.2.218.62:33281 was elected leader, sending a full tablet report...
I20260923 14:39:58.161581 20463 log.cc:824] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:58.161779 20459 ts_tablet_manager.cc:1428] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260923 14:39:58.163888 20463 tablet_bootstrap.cc:492] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: No bootstrap required, opened a new log
I20260923 14:39:58.164184 20463 ts_tablet_manager.cc:1397] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 14:39:58.166697 20463 raft_consensus.cc:348] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 } } peers { permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 } } peers { permanent_uuid: "a9788b4256094d8a8085ed0443e0075d" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40415 } }
I20260923 14:39:58.166867 20463 raft_consensus.cc:374] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:58.166898 20463 raft_consensus.cc:729] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9788b4256094d8a8085ed0443e0075d, State: Initialized, Role: FOLLOWER
I20260923 14:39:58.167161 20463 consensus_queue.cc:260] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 } } peers { permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 } } peers { permanent_uuid: "a9788b4256094d8a8085ed0443e0075d" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40415 } }
I20260923 14:39:58.167629 20308 heartbeater.cc:503] Master 127.2.218.62:33281 was elected leader, sending a full tablet report...
I20260923 14:39:58.167832 20463 ts_tablet_manager.cc:1428] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260923 14:39:58.217329 20175 tablet.cc:2369] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:39:58.336004 20467 raft_consensus.cc:482] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:39:58.336222 20467 raft_consensus.cc:504] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 } } peers { permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 } } peers { permanent_uuid: "a9788b4256094d8a8085ed0443e0075d" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40415 } }
I20260923 14:39:58.337224 20467 leader_election.cc:288] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56f1205eb08342ec92bfaf4c04b1fa42 (127.2.218.3:36465), ab9d60a7c4394910a8bd0ba3697ddefe (127.2.218.1:36607)
I20260923 14:39:58.340731 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.337473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:36465 (local address 127.2.218.2:50691)
0923 14:39:58.337560 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:58.337573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:58.337648 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:58.338123 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:58.338128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:58.338148 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:58.338271 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.338283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.338942 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.338946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.339707 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.339715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.340283 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.340286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.340451 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.340466 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 14:39:58.340546 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:39:58.340614 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":96}
I20260923 14:39:58.340721 20457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.337557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:49953 (local address 127.2.218.1:36607)
0923 14:39:58.337659 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.337663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.337950 (+ 287us) server_negotiation.cc:407] Connection header received
0923 14:39:58.338031 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.338032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.338090 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.338174 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:58.338428 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.338837 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.339854 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.340180 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.340280 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.340339 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.340549 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.340611 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 14:39:58.340737 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.337752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:50691 (local address 127.2.218.3:36465)
0923 14:39:58.337884 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.337889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.337904 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:39:58.337952 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.337955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.338005 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.338074 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:58.338404 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.338839 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.339829 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.340178 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.340282 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.340342 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.340583 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.340645 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 14:39:58.340750 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.337473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:36607 (local address 127.2.218.2:49953)
0923 14:39:58.337854 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:58.337872 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:58.337962 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:58.338199 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:58.338201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:58.338221 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:58.338309 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.338313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.338948 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.338951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.339714 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.339722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.340289 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.340291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.340416 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.340455 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 14:39:58.340507 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 14:39:58.340593 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":83,"thread_start_us":112,"threads_started":1}
W20260923 14:39:58.341346 20443 tablet.cc:2369] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:39:58.341310 20396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf468ddb9be4c0fa768da1990276acd" candidate_uuid: "a9788b4256094d8a8085ed0443e0075d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" is_pre_election: true
I20260923 14:39:58.341415 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf468ddb9be4c0fa768da1990276acd" candidate_uuid: "a9788b4256094d8a8085ed0443e0075d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" is_pre_election: true
I20260923 14:39:58.341595 20396 raft_consensus.cc:2454] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9788b4256094d8a8085ed0443e0075d in term 0.
I20260923 14:39:58.341650 20128 raft_consensus.cc:2454] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9788b4256094d8a8085ed0443e0075d in term 0.
I20260923 14:39:58.342025 20198 leader_election.cc:302] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9788b4256094d8a8085ed0443e0075d, ab9d60a7c4394910a8bd0ba3697ddefe; no voters:
I20260923 14:39:58.342275 20467 raft_consensus.cc:2781] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:39:58.342350 20467 raft_consensus.cc:482] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:39:58.342381 20467 raft_consensus.cc:3037] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:58.343272 20467 raft_consensus.cc:504] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 } } peers { permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 } } peers { permanent_uuid: "a9788b4256094d8a8085ed0443e0075d" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40415 } }
I20260923 14:39:58.343565 20467 leader_election.cc:288] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [CANDIDATE]: Term 1 election: Requested vote from peers 56f1205eb08342ec92bfaf4c04b1fa42 (127.2.218.3:36465), ab9d60a7c4394910a8bd0ba3697ddefe (127.2.218.1:36607)
I20260923 14:39:58.343819 20396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf468ddb9be4c0fa768da1990276acd" candidate_uuid: "a9788b4256094d8a8085ed0443e0075d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56f1205eb08342ec92bfaf4c04b1fa42"
I20260923 14:39:58.343912 20396 raft_consensus.cc:3037] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:58.344041 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cf468ddb9be4c0fa768da1990276acd" candidate_uuid: "a9788b4256094d8a8085ed0443e0075d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe"
I20260923 14:39:58.344165 20128 raft_consensus.cc:3037] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:58.344864 20396 raft_consensus.cc:2454] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9788b4256094d8a8085ed0443e0075d in term 1.
I20260923 14:39:58.345112 20196 leader_election.cc:302] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56f1205eb08342ec92bfaf4c04b1fa42, a9788b4256094d8a8085ed0443e0075d; no voters:
I20260923 14:39:58.345187 20128 raft_consensus.cc:2454] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9788b4256094d8a8085ed0443e0075d in term 1.
I20260923 14:39:58.345257 20467 raft_consensus.cc:2781] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:39:58.345445 20467 raft_consensus.cc:686] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 1 LEADER]: Becoming Leader. State: Replica: a9788b4256094d8a8085ed0443e0075d, State: Running, Role: LEADER
I20260923 14:39:58.345757 20467 consensus_queue.cc:237] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 } } peers { permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 } } peers { permanent_uuid: "a9788b4256094d8a8085ed0443e0075d" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40415 } }
I20260923 14:39:58.348668 19983 catalog_manager.cc:5495] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d reported cstate change: term changed from 0 to 1, leader changed from <none> to a9788b4256094d8a8085ed0443e0075d (127.2.218.2). New cstate: current_term: 1 leader_uuid: "a9788b4256094d8a8085ed0443e0075d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9788b4256094d8a8085ed0443e0075d" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40415 } health_report { overall_health: HEALTHY } } }
I20260923 14:39:58.370405 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.367443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.2.218.2:40415)
0923 14:39:58.367554 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.367557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.367650 (+ 93us) server_negotiation.cc:407] Connection header received
0923 14:39:58.367770 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.367772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.367810 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.367882 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:39:58.368069 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.368462 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.369141 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.369305 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.369377 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.369625 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:39:58.369960 (+ 335us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:39:58.370041 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.370189 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.370231 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:39:58.374912 20396 raft_consensus.cc:1261] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42 [term 1 FOLLOWER]: Refusing update from remote peer a9788b4256094d8a8085ed0443e0075d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:39:58.374927 20128 raft_consensus.cc:1261] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe [term 1 FOLLOWER]: Refusing update from remote peer a9788b4256094d8a8085ed0443e0075d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:39:58.375458 20467 consensus_queue.cc:1035] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9d60a7c4394910a8bd0ba3697ddefe" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:39:58.375614 20469 consensus_queue.cc:1035] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [LEADER]: Connected to new peer: Peer: permanent_uuid: "56f1205eb08342ec92bfaf4c04b1fa42" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:39:58.381129 20472 mvcc.cc:205] Tried to move back new op lower bound from 7332554335737802752 to 7332554335626579968. Current Snapshot: MvccSnapshot[applied={T|T < 7332554335737802752}]
I20260923 14:39:58.382076 20473 mvcc.cc:205] Tried to move back new op lower bound from 7332554335737802752 to 7332554335626579968. Current Snapshot: MvccSnapshot[applied={T|T < 7332554335737802752}]
I20260923 14:39:58.384909 20475 mvcc.cc:205] Tried to move back new op lower bound from 7332554335737802752 to 7332554335626579968. Current Snapshot: MvccSnapshot[applied={T|T < 7332554335737802752}]
I20260923 14:39:58.388365 19983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:39:58.388506 19983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:39:58.390395 19983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5591621200e14538aa677998bbca1582: Sending DeleteTablet for 3 replicas of tablet 0cf468ddb9be4c0fa768da1990276acd
I20260923 14:39:58.390997 20376 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf468ddb9be4c0fa768da1990276acd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:39:58 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:34804
I20260923 14:39:58.391095 20108 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf468ddb9be4c0fa768da1990276acd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:39:58 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:48684
I20260923 14:39:58.391084 20242 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cf468ddb9be4c0fa768da1990276acd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:39:58 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:47534
I20260923 14:39:58.391418 20482 tablet_replica.cc:330] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: stopping tablet replica
I20260923 14:39:58.391577 20483 tablet_replica.cc:330] T 0cf468ddb9be4c0fa768da1990276acd P ab9d60a7c4394910a8bd0ba3697ddefe: stopping tablet replica
I20260923 14:39:58.391693 20482 raft_consensus.cc:2229] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:39:58.391820 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 20045
I20260923 14:39:58.394413 20482 raft_consensus.cc:2258] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:39:58.396126 20482 ts_tablet_manager.cc:1905] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:39:58.397723 20484 tablet_replica.cc:330] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: stopping tablet replica
I20260923 14:39:58.397959 20484 raft_consensus.cc:2229] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:39:58.398252 20484 raft_consensus.cc:2258] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:39:58.398474 20482 ts_tablet_manager.cc:1918] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260923 14:39:58.398548 19970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:36607 (error 108)
I20260923 14:39:58.398587 20482 log.cc:1196] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/0cf468ddb9be4c0fa768da1990276acd
I20260923 14:39:58.398869 20482 ts_tablet_manager.cc:1939] T 0cf468ddb9be4c0fa768da1990276acd P 56f1205eb08342ec92bfaf4c04b1fa42: Deleting consensus metadata
I20260923 14:39:58.399845 19968 catalog_manager.cc:4841] TS 56f1205eb08342ec92bfaf4c04b1fa42 (127.2.218.3:36465): tablet 0cf468ddb9be4c0fa768da1990276acd (table test-table [id=9ad7b601662649ad8b9a88babc3d65ab]) successfully deleted
I20260923 14:39:58.399986 20484 ts_tablet_manager.cc:1905] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:39:58.401665 20484 ts_tablet_manager.cc:1918] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:39:58.401757 20484 log.cc:1196] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/0cf468ddb9be4c0fa768da1990276acd
I20260923 14:39:58.401986 20484 ts_tablet_manager.cc:1939] T 0cf468ddb9be4c0fa768da1990276acd P a9788b4256094d8a8085ed0443e0075d: Deleting consensus metadata
W20260923 14:39:58.402917 20456 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:39:58.399200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:36607 (local address 127.0.0.1:48694)
0923 14:39:58.399307 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:58.399322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:58.399397 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:58.402822 (+ 3425us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:36607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 14:39:58.403059 19970 catalog_manager.cc:4584] TS ab9d60a7c4394910a8bd0ba3697ddefe (127.2.218.1:36607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0cf468ddb9be4c0fa768da1990276acd: Network error: Client connection negotiation failed: client connection to 127.2.218.1:36607: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 14:39:58.403096 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 20179
I20260923 14:39:58.403226 19970 catalog_manager.cc:4841] TS a9788b4256094d8a8085ed0443e0075d (127.2.218.2:40415): tablet 0cf468ddb9be4c0fa768da1990276acd (table test-table [id=9ad7b601662649ad8b9a88babc3d65ab]) successfully deleted
I20260923 14:39:58.408223 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 20313
I20260923 14:39:58.412940 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 19951
2026-09-23T14:39:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2146 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T14:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:39:58Z Disabled control of system clock
I20260923 14:39:58.440549 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:44575
--webserver_interface=127.2.218.62
--webserver_port=0
--builtin_ntp_servers=127.2.218.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:39:58.533201 20489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:58.533447 20489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:58.533519 20489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:58.536911 20489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:39:58.536995 20489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:58.537022 20489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:58.537073 20489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:39:58.537108 20489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:39:58.541059 20489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:44575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:58.541853 20489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:58.542960 20489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:58.550132 20494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:58.550136 20497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:58.550235 20495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:58.550230 20489 server_base.cc:1034] running on GCE node
I20260923 14:39:58.550872 20489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:58.551748 20489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:58.552928 20489 hybrid_clock.cc:648] HybridClock initialized: now 1790174398552895 us; error 69 us; skew 500 ppm
I20260923 14:39:58.554610 20489 webserver.cc:460] Webserver started at http://127.2.218.62:33055/ using document root <none> and password file <none>
I20260923 14:39:58.555090 20489 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:58.555181 20489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:58.555401 20489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:39:58.557070 20489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "6d41818d2b1d4b2c89bc59b2dbe28390"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "5fc5bfd9d317a984efca5b747a19550d"
server_key_iv: "cfebf04ca0d792ac8f8ca67e5e0b2d26"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.557590 20489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "6d41818d2b1d4b2c89bc59b2dbe28390"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "5fc5bfd9d317a984efca5b747a19550d"
server_key_iv: "cfebf04ca0d792ac8f8ca67e5e0b2d26"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.560721 20489 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:39:58.562721 20503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:58.563695 20489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:58.563839 20489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "6d41818d2b1d4b2c89bc59b2dbe28390"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "5fc5bfd9d317a984efca5b747a19550d"
server_key_iv: "cfebf04ca0d792ac8f8ca67e5e0b2d26"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.563948 20489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:58.579651 20489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:58.580183 20489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:58.580369 20489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:58.586619 20489 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:44575
I20260923 14:39:58.586639 20555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:44575 every 8 connection(s)
I20260923 14:39:58.587390 20489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:39:58.589780 20556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:58.593937 20556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390: Bootstrap starting.
I20260923 14:39:58.594753 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 20489
I20260923 14:39:58.594878 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:39:58.595111 2920 external_mini_cluster.cc:1428] Setting key 75ef95f3f93d83aec5e0715e50337f27
I20260923 14:39:58.596112 20556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:58.596755 20556 log.cc:824] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:58.598493 20556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390: No bootstrap required, opened a new log
I20260923 14:39:58.601327 20556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41818d2b1d4b2c89bc59b2dbe28390" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44575 } }
I20260923 14:39:58.601492 20556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:58.601564 20556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d41818d2b1d4b2c89bc59b2dbe28390, State: Initialized, Role: FOLLOWER
I20260923 14:39:58.601917 20556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41818d2b1d4b2c89bc59b2dbe28390" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44575 } }
I20260923 14:39:58.602037 20556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:39:58.602152 20556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:39:58.602277 20556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:58.603014 20556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41818d2b1d4b2c89bc59b2dbe28390" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44575 } }
I20260923 14:39:58.603202 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.596077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59088 (local address 127.2.218.62:44575)
0923 14:39:58.596604 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.596619 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.596647 (+ 28us) server_negotiation.cc:407] Connection header received
0923 14:39:58.597330 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.597350 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.597706 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.598020 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:58.598440 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.599344 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.600135 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.600540 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.600701 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.601482 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:58.601489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:58.601503 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:39:58.601949 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:58.601999 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:39:58.602039 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:58.602169 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.602336 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.602576 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
I20260923 14:39:58.603323 20556 leader_election.cc:302] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d41818d2b1d4b2c89bc59b2dbe28390; no voters:
I20260923 14:39:58.603605 20556 leader_election.cc:288] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:39:58.603708 20561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:39:58.603914 20561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [term 1 LEADER]: Becoming Leader. State: Replica: 6d41818d2b1d4b2c89bc59b2dbe28390, State: Running, Role: LEADER
I20260923 14:39:58.604249 20561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41818d2b1d4b2c89bc59b2dbe28390" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44575 } }
I20260923 14:39:58.604651 20556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:39:58.605720 20563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d41818d2b1d4b2c89bc59b2dbe28390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41818d2b1d4b2c89bc59b2dbe28390" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44575 } } }
I20260923 14:39:58.605834 20563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [sys.catalog]: This master's current role is: LEADER
I20260923 14:39:58.606158 20562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d41818d2b1d4b2c89bc59b2dbe28390. Latest consensus state: current_term: 1 leader_uuid: "6d41818d2b1d4b2c89bc59b2dbe28390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41818d2b1d4b2c89bc59b2dbe28390" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 44575 } } }
I20260923 14:39:58.606215 20570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:39:58.606300 20562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390 [sys.catalog]: This master's current role is: LEADER
I20260923 14:39:58.608584 20570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:39:58.613042 20570 catalog_manager.cc:1347] Generated new cluster ID: 42c47d73e7b84d5f8aea2e1e9dffb661
I20260923 14:39:58.613103 20570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:39:58.642310 20570 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:39:58.643003 20570 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:39:58.647441 20570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390: Generated new TSK 0
I20260923 14:39:58.647933 20570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:39:58.654033 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44575
--builtin_ntp_servers=127.2.218.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:39:58.749511 20580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:58.749696 20580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:58.749725 20580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:58.753001 20580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:58.753039 20580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:58.753082 20580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:39:58.756881 20580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:58.757563 20580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:58.758644 20580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:58.766208 20585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:58.766328 20580 server_base.cc:1034] running on GCE node
W20260923 14:39:58.766207 20586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:58.766495 20588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:58.766824 20580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:58.767310 20580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:58.768460 20580 hybrid_clock.cc:648] HybridClock initialized: now 1790174398768452 us; error 23 us; skew 500 ppm
I20260923 14:39:58.769843 20580 webserver.cc:460] Webserver started at http://127.2.218.1:39211/ using document root <none> and password file <none>
I20260923 14:39:58.770349 20580 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:58.770408 20580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:58.770603 20580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:39:58.772188 20580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "2145556be0924062806249e81a671b9f"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "dafa3eb2714a3fab2804912a6ebf6803"
server_key_iv: "dafd4f63acfadb315ee66afc81894471"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.772641 20580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "2145556be0924062806249e81a671b9f"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "dafa3eb2714a3fab2804912a6ebf6803"
server_key_iv: "dafd4f63acfadb315ee66afc81894471"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.775720 20580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:39:58.777660 20594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:58.778702 20580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:58.778810 20580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "2145556be0924062806249e81a671b9f"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "dafa3eb2714a3fab2804912a6ebf6803"
server_key_iv: "dafd4f63acfadb315ee66afc81894471"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.778908 20580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:58.791646 20580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:58.792119 20580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:58.792273 20580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:58.792709 20580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:58.793422 20580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:39:58.793483 20580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:58.793537 20580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:39:58.793579 20580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:58.801419 20580 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:35747
I20260923 14:39:58.801442 20707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:35747 every 8 connection(s)
I20260923 14:39:58.802136 20580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:39:58.808945 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 20580
I20260923 14:39:58.809046 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:39:58.809289 2920 external_mini_cluster.cc:1428] Setting key f0d014985b601581022ebb0044954229
I20260923 14:39:58.809793 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.803683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:35895 (local address 127.2.218.62:44575)
0923 14:39:58.803803 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.803813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.804645 (+ 832us) server_negotiation.cc:407] Connection header received
0923 14:39:58.805903 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.805906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.805959 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.806048 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:58.806795 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.807258 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.808202 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.808401 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.808499 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.809375 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:58.809379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:58.809381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:39:58.809392 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:58.809414 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:39:58.809422 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:58.809491 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.809624 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.809674 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:39:58.810528 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.803974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:44575 (local address 127.2.218.1:35895)
0923 14:39:58.804505 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:58.804536 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:58.805704 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:58.806204 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:58.806210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:58.806301 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:58.806670 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.806679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.807385 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.807388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.808091 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.808099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.808528 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.808531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.808665 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.809168 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:39:58.809170 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:58.809198 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:39:58.809245 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:58.809492 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:58.809502 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:58.809508 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 14:39:58.809622 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 14:39:58.809829 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260923 14:39:58.811115 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44575
--builtin_ntp_servers=127.2.218.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:39:58.811534 20708 heartbeater.cc:347] Connected to a master server at 127.2.218.62:44575
I20260923 14:39:58.811733 20708 heartbeater.cc:464] Registering TS with master...
I20260923 14:39:58.812110 20708 heartbeater.cc:511] Master 127.2.218.62:44575 requested a full tablet report, sending...
I20260923 14:39:58.813319 20520 ts_manager.cc:194] Registered new tserver with Master: 2145556be0924062806249e81a671b9f (127.2.218.1:35747)
I20260923 14:39:58.814401 20520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.1:35895
W20260923 14:39:58.900846 20711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:58.901015 20711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:58.901044 20711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:58.904907 20711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:58.904948 20711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:58.905028 20711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:39:58.909161 20711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:58.909854 20711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:58.910884 20711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:58.918349 20719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:58.918339 20717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:58.918370 20716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:58.918606 20711 server_base.cc:1034] running on GCE node
I20260923 14:39:58.918968 20711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:58.919425 20711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:58.920588 20711 hybrid_clock.cc:648] HybridClock initialized: now 1790174398920568 us; error 29 us; skew 500 ppm
I20260923 14:39:58.922075 20711 webserver.cc:460] Webserver started at http://127.2.218.2:44701/ using document root <none> and password file <none>
I20260923 14:39:58.922583 20711 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:58.922641 20711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:58.922851 20711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:39:58.924428 20711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "b4c5a4513d39453c9d6c12e56c3acc1a"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "eab1c2f620bc5a21c841d4a12ab696b2"
server_key_iv: "e1a08a30f7d792137488a83573def5ca"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.924917 20711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "b4c5a4513d39453c9d6c12e56c3acc1a"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "eab1c2f620bc5a21c841d4a12ab696b2"
server_key_iv: "e1a08a30f7d792137488a83573def5ca"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.928021 20711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:39:58.930008 20725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:58.930977 20711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:58.931104 20711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "b4c5a4513d39453c9d6c12e56c3acc1a"
format_stamp: "Formatted at 2026-09-23 14:39:58 on dist-test-slave-zl47"
server_key: "eab1c2f620bc5a21c841d4a12ab696b2"
server_key_iv: "e1a08a30f7d792137488a83573def5ca"
server_key_version: "encryptionkey@0"
I20260923 14:39:58.931205 20711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:58.955595 20711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:58.956166 20711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:58.956346 20711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:58.956818 20711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:58.957614 20711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:39:58.957685 20711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:58.957748 20711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:39:58.957795 20711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:58.966207 20711 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:40529
I20260923 14:39:58.966228 20838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:40529 every 8 connection(s)
I20260923 14:39:58.966926 20711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:39:58.973958 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.968376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:58839 (local address 127.2.218.62:44575)
0923 14:39:58.968491 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:58.968495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:58.969247 (+ 752us) server_negotiation.cc:407] Connection header received
0923 14:39:58.970329 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:58.970331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:58.970372 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:58.970437 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:58.971112 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.971545 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.972434 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.972612 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.972679 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.973595 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:58.973597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:58.973599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:39:58.973608 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:58.973622 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:39:58.973628 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:58.973680 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:58.973806 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 14:39:58.973850 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:39:58.974677 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:58.968656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:44575 (local address 127.2.218.2:58839)
0923 14:39:58.969150 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:58.969181 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:58.970178 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:58.970568 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:58.970575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:58.970641 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:58.971008 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.971018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.971632 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.971634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.972340 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:58.972348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:58.972719 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:58.972721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:58.972834 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:58.973382 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:39:58.973384 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:58.973418 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:39:58.973477 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:58.973698 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:58.973704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:58.973709 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 14:39:58.973792 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 14:39:58.973983 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260923 14:39:58.975548 20839 heartbeater.cc:347] Connected to a master server at 127.2.218.62:44575
I20260923 14:39:58.975737 20839 heartbeater.cc:464] Registering TS with master...
I20260923 14:39:58.976118 20839 heartbeater.cc:511] Master 127.2.218.62:44575 requested a full tablet report, sending...
I20260923 14:39:58.976317 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 20711
I20260923 14:39:58.976442 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:39:58.976720 2920 external_mini_cluster.cc:1428] Setting key c09be8dc0a96700be26bfe8b009cbc98
I20260923 14:39:58.977079 20520 ts_manager.cc:194] Registered new tserver with Master: b4c5a4513d39453c9d6c12e56c3acc1a (127.2.218.2:40529)
I20260923 14:39:58.977767 20520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.2:58839
I20260923 14:39:58.978761 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44575
--builtin_ntp_servers=127.2.218.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:39:59.069069 20842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:39:59.069250 20842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:39:59.069279 20842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:39:59.072688 20842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:39:59.072729 20842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:39:59.072775 20842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:39:59.076824 20842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:39:59.077644 20842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:39:59.078737 20842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:39:59.086591 20850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:59.086637 20847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:39:59.086591 20848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:39:59.086629 20842 server_base.cc:1034] running on GCE node
I20260923 14:39:59.087123 20842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:39:59.087626 20842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:39:59.088779 20842 hybrid_clock.cc:648] HybridClock initialized: now 1790174399088749 us; error 39 us; skew 500 ppm
I20260923 14:39:59.090408 20842 webserver.cc:460] Webserver started at http://127.2.218.3:46503/ using document root <none> and password file <none>
I20260923 14:39:59.090873 20842 fs_manager.cc:359] Metadata directory not provided
I20260923 14:39:59.090931 20842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:39:59.091147 20842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:39:59.092767 20842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "52f999473a364119a4b5902bc646512a"
format_stamp: "Formatted at 2026-09-23 14:39:59 on dist-test-slave-zl47"
server_key: "9f3590124c6e77f5cbdfb1c5a108d049"
server_key_iv: "73d5f7ddc93f6a0f54da444eb0798952"
server_key_version: "encryptionkey@0"
I20260923 14:39:59.093273 20842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "52f999473a364119a4b5902bc646512a"
format_stamp: "Formatted at 2026-09-23 14:39:59 on dist-test-slave-zl47"
server_key: "9f3590124c6e77f5cbdfb1c5a108d049"
server_key_iv: "73d5f7ddc93f6a0f54da444eb0798952"
server_key_version: "encryptionkey@0"
I20260923 14:39:59.096369 20842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260923 14:39:59.098452 20856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:59.099517 20842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:39:59.099632 20842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "52f999473a364119a4b5902bc646512a"
format_stamp: "Formatted at 2026-09-23 14:39:59 on dist-test-slave-zl47"
server_key: "9f3590124c6e77f5cbdfb1c5a108d049"
server_key_iv: "73d5f7ddc93f6a0f54da444eb0798952"
server_key_version: "encryptionkey@0"
I20260923 14:39:59.099722 20842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:39:59.117022 20842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:39:59.117494 20842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:39:59.117645 20842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:39:59.118081 20842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:39:59.118846 20842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:39:59.118885 20842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:59.118939 20842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:39:59.118970 20842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:39:59.126292 20842 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:40811
I20260923 14:39:59.126317 20969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:40811 every 8 connection(s)
I20260923 14:39:59.127000 20842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:39:59.133699 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 20842
I20260923 14:39:59.133812 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:39:59.134066 2920 external_mini_cluster.cc:1428] Setting key b51fba3866445ddfe1f59bef8b22fa63
I20260923 14:39:59.134128 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.128358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:50457 (local address 127.2.218.62:44575)
0923 14:39:59.128505 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:59.128510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:59.129286 (+ 776us) server_negotiation.cc:407] Connection header received
0923 14:39:59.130401 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:59.130403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:59.130442 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:59.130506 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:59.131172 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.131600 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.132471 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.132647 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.132710 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.133624 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:59.133628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:59.133630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:39:59.133642 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:59.133661 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:39:59.133670 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:59.133736 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:59.133913 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 14:39:59.133966 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 14:39:59.134995 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.128678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:44575 (local address 127.2.218.3:50457)
0923 14:39:59.129168 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:59.129198 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:59.130250 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:59.130635 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:59.130640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:59.130705 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:39:59.131069 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.131078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.131689 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.131691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.132384 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.132392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.132752 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.132755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.132866 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.133384 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:39:59.133386 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:39:59.133416 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:39:59.133471 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:39:59.133746 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:39:59.133767 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:39:59.133774 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 14:39:59.133899 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 14:39:59.134085 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260923 14:39:59.136003 20970 heartbeater.cc:347] Connected to a master server at 127.2.218.62:44575
I20260923 14:39:59.136178 20970 heartbeater.cc:464] Registering TS with master...
I20260923 14:39:59.136524 20970 heartbeater.cc:511] Master 127.2.218.62:44575 requested a full tablet report, sending...
I20260923 14:39:59.137264 20520 ts_manager.cc:194] Registered new tserver with Master: 52f999473a364119a4b5902bc646512a (127.2.218.3:40811)
I20260923 14:39:59.137723 20520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.3:50457
I20260923 14:39:59.147501 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:39:59.152333 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.149111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.2.218.62:44575)
0923 14:39:59.149221 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:59.149225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:59.149390 (+ 165us) server_negotiation.cc:407] Connection header received
0923 14:39:59.149513 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:59.149515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:59.149547 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:59.149624 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:39:59.149872 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.150395 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.151141 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.151333 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.151426 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.152004 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:39:59.152007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:39:59.152015 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:39:59.152024 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:39:59.152037 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:39:59.152043 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:39:59.152114 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:59.152187 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 14:39:59.152228 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:39:59.154417 20520 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:39:59.156934 20520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:39:59.168880 20981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.163679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40529 (local address 127.0.0.1:36706)
0923 14:39:59.164282 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:59.164325 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:59.164427 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:59.165055 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:59.165059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:59.165077 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:59.165177 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.165202 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.166563 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.166568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.167696 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.167703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.168442 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.168445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.168541 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.168572 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 14:39:59.168666 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 14:39:59.168757 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":74,"spinlock_wait_cycles":1280,"thread_start_us":65,"threads_started":1}
I20260923 14:39:59.168891 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.163817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.2.218.2:40529)
0923 14:39:59.164687 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:59.164697 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:59.164715 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:39:59.164775 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:59.164779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:59.164933 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:59.165059 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:59.165318 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.166462 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.167813 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.168329 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.168442 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.168619 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:59.168692 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 14:39:59.168768 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":103,"threads_started":1}
I20260923 14:39:59.169142 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.163962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:35747 (local address 127.0.0.1:36356)
0923 14:39:59.164281 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:59.164323 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:59.164426 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:59.165219 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:59.165224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:59.165244 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:59.165334 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.165339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.166606 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.166609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.167915 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.167921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.168812 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.168816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.168899 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.168919 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:39:59.168963 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 14:39:59.169019 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":211,"spinlock_wait_cycles":1408,"thread_start_us":81,"threads_started":1}
I20260923 14:39:59.169454 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.164233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:40811 (local address 127.0.0.1:52688)
0923 14:39:59.164933 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:59.164945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:59.165016 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:59.165457 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:59.165460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:59.165476 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:59.165567 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.165579 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.166595 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.166597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.167903 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.167914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.169115 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.169118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.169198 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.169222 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 14:39:59.169282 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 14:39:59.169346 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":139,"thread_start_us":74,"threads_started":1}
I20260923 14:39:59.169806 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.164546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.2.218.3:40811)
0923 14:39:59.164769 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:59.164782 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:59.165108 (+ 326us) server_negotiation.cc:407] Connection header received
0923 14:39:59.165165 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:59.165170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:59.165347 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:59.165475 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:59.165672 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.166450 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.168035 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.168690 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.168791 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.169555 (+ 764us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:59.169600 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 14:39:59.169672 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260923 14:39:59.169803 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.164106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.2.218.1:35747)
0923 14:39:59.164472 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:59.164484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:59.164506 (+ 22us) server_negotiation.cc:407] Connection header received
0923 14:39:59.164575 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:59.164579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:59.164753 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:59.164880 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:59.165449 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.166491 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.168034 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.168691 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.169380 (+ 689us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.169551 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:59.169599 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 14:39:59.169673 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":80,"threads_started":1}
I20260923 14:39:59.171532 20773 tablet_service.cc:1467] Processing CreateTablet for tablet 0032320490a14a41af21d08c0712bb59 (DEFAULT_TABLE table=test-table [id=3802249d82ac472b89abc575f8efbcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:39:59.172088 20904 tablet_service.cc:1467] Processing CreateTablet for tablet 0032320490a14a41af21d08c0712bb59 (DEFAULT_TABLE table=test-table [id=3802249d82ac472b89abc575f8efbcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:39:59.172501 20773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0032320490a14a41af21d08c0712bb59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:59.172951 20904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0032320490a14a41af21d08c0712bb59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:59.175704 20642 tablet_service.cc:1467] Processing CreateTablet for tablet 0032320490a14a41af21d08c0712bb59 (DEFAULT_TABLE table=test-table [id=3802249d82ac472b89abc575f8efbcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:39:59.176441 20642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0032320490a14a41af21d08c0712bb59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:39:59.179670 20987 tablet_bootstrap.cc:492] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: Bootstrap starting.
I20260923 14:39:59.181715 20987 tablet_bootstrap.cc:654] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:59.181993 20989 tablet_bootstrap.cc:492] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f: Bootstrap starting.
I20260923 14:39:59.182349 20988 tablet_bootstrap.cc:492] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: Bootstrap starting.
I20260923 14:39:59.183482 20989 tablet_bootstrap.cc:654] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:59.183810 20988 tablet_bootstrap.cc:654] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: Neither blocks nor log segments found. Creating new log.
I20260923 14:39:59.184051 20987 log.cc:824] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:59.184104 20989 log.cc:824] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:59.184625 20988 log.cc:824] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:39:59.185781 20987 tablet_bootstrap.cc:492] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: No bootstrap required, opened a new log
I20260923 14:39:59.185943 20987 ts_tablet_manager.cc:1397] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 14:39:59.185993 20989 tablet_bootstrap.cc:492] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f: No bootstrap required, opened a new log
I20260923 14:39:59.186084 20988 tablet_bootstrap.cc:492] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: No bootstrap required, opened a new log
I20260923 14:39:59.186152 20989 ts_tablet_manager.cc:1397] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260923 14:39:59.186275 20988 ts_tablet_manager.cc:1397] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260923 14:39:59.188767 20988 raft_consensus.cc:348] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 } } peers { permanent_uuid: "52f999473a364119a4b5902bc646512a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 40811 } } peers { permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 } }
I20260923 14:39:59.188972 20988 raft_consensus.cc:374] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:59.189018 20988 raft_consensus.cc:729] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f999473a364119a4b5902bc646512a, State: Initialized, Role: FOLLOWER
I20260923 14:39:59.189401 20988 consensus_queue.cc:260] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 } } peers { permanent_uuid: "52f999473a364119a4b5902bc646512a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 40811 } } peers { permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 } }
I20260923 14:39:59.189576 20987 raft_consensus.cc:348] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 } } peers { permanent_uuid: "52f999473a364119a4b5902bc646512a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 40811 } } peers { permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 } }
I20260923 14:39:59.189585 20989 raft_consensus.cc:348] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 } } peers { permanent_uuid: "52f999473a364119a4b5902bc646512a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 40811 } } peers { permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 } }
I20260923 14:39:59.189780 20987 raft_consensus.cc:374] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:59.189780 20989 raft_consensus.cc:374] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:39:59.189829 20989 raft_consensus.cc:729] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2145556be0924062806249e81a671b9f, State: Initialized, Role: FOLLOWER
I20260923 14:39:59.189836 20987 raft_consensus.cc:729] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c5a4513d39453c9d6c12e56c3acc1a, State: Initialized, Role: FOLLOWER
I20260923 14:39:59.190012 20970 heartbeater.cc:503] Master 127.2.218.62:44575 was elected leader, sending a full tablet report...
I20260923 14:39:59.190240 20988 ts_tablet_manager.cc:1428] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260923 14:39:59.190203 20987 consensus_queue.cc:260] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 } } peers { permanent_uuid: "52f999473a364119a4b5902bc646512a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 40811 } } peers { permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 } }
I20260923 14:39:59.190203 20989 consensus_queue.cc:260] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 } } peers { permanent_uuid: "52f999473a364119a4b5902bc646512a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 40811 } } peers { permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 } }
I20260923 14:39:59.190905 20839 heartbeater.cc:503] Master 127.2.218.62:44575 was elected leader, sending a full tablet report...
I20260923 14:39:59.190961 20987 ts_tablet_manager.cc:1428] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:39:59.191324 20989 ts_tablet_manager.cc:1428] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:39:59.191772 20708 heartbeater.cc:503] Master 127.2.218.62:44575 was elected leader, sending a full tablet report...
W20260923 14:39:59.217724 20840 tablet.cc:2369] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:39:59.224763 20993 raft_consensus.cc:482] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:39:59.224874 20993 raft_consensus.cc:504] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 } } peers { permanent_uuid: "52f999473a364119a4b5902bc646512a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 40811 } } peers { permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 } }
I20260923 14:39:59.225690 20993 leader_election.cc:288] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4c5a4513d39453c9d6c12e56c3acc1a (127.2.218.2:40529), 2145556be0924062806249e81a671b9f (127.2.218.1:35747)
I20260923 14:39:59.228614 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.225907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:35099 (local address 127.2.218.2:40529)
0923 14:39:59.226009 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:59.226015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:59.226030 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:39:59.226162 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:59.226165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:59.226206 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:59.226282 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:59.226520 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.226931 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.227785 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.228134 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.228244 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.228294 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:59.228460 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 14:39:59.228523 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:39:59.228614 20972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.225827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40529 (local address 127.2.218.3:35099)
0923 14:39:59.225946 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:59.225961 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:59.226031 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:59.226296 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:59.226299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:59.226314 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:59.226420 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.226425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.227034 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.227037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.227674 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.227682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.228247 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.228251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.228344 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.228365 (+ 21us) client_negotiation.cc:769] Sending connection context
0923 14:39:59.228445 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:39:59.228507 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 14:39:59.228843 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.226026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:60231 (local address 127.2.218.1:35747)
0923 14:39:59.226139 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:59.226142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:59.226284 (+ 142us) server_negotiation.cc:407] Connection header received
0923 14:39:59.226405 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:59.226408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:59.226446 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:59.226517 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:39:59.226748 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.227144 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.227995 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.228369 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.228477 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.228522 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:59.228697 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 14:39:59.228752 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 14:39:59.228859 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.225907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:35747 (local address 127.2.218.3:60231)
0923 14:39:59.226196 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:59.226206 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:39:59.226311 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:39:59.226526 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:39:59.226529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:39:59.226542 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:39:59.226653 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.226657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.227248 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.227252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.227886 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:39:59.227893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.228480 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.228484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:39:59.228588 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.228604 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 14:39:59.228682 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:39:59.228740 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260923 14:39:59.229132 20793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0032320490a14a41af21d08c0712bb59" candidate_uuid: "52f999473a364119a4b5902bc646512a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" is_pre_election: true
I20260923 14:39:59.229379 20793 raft_consensus.cc:2454] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f999473a364119a4b5902bc646512a in term 0.
I20260923 14:39:59.229458 20662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0032320490a14a41af21d08c0712bb59" candidate_uuid: "52f999473a364119a4b5902bc646512a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145556be0924062806249e81a671b9f" is_pre_election: true
I20260923 14:39:59.229691 20662 raft_consensus.cc:2454] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f999473a364119a4b5902bc646512a in term 0.
I20260923 14:39:59.229828 20859 leader_election.cc:302] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f999473a364119a4b5902bc646512a, b4c5a4513d39453c9d6c12e56c3acc1a; no voters:
I20260923 14:39:59.230034 20993 raft_consensus.cc:2781] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:39:59.230152 20993 raft_consensus.cc:482] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:39:59.230206 20993 raft_consensus.cc:3037] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:59.230968 20993 raft_consensus.cc:504] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 } } peers { permanent_uuid: "52f999473a364119a4b5902bc646512a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 40811 } } peers { permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 } }
I20260923 14:39:59.231268 20993 leader_election.cc:288] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [CANDIDATE]: Term 1 election: Requested vote from peers b4c5a4513d39453c9d6c12e56c3acc1a (127.2.218.2:40529), 2145556be0924062806249e81a671b9f (127.2.218.1:35747)
I20260923 14:39:59.231510 20793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0032320490a14a41af21d08c0712bb59" candidate_uuid: "52f999473a364119a4b5902bc646512a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a"
I20260923 14:39:59.231571 20662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0032320490a14a41af21d08c0712bb59" candidate_uuid: "52f999473a364119a4b5902bc646512a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2145556be0924062806249e81a671b9f"
I20260923 14:39:59.231604 20793 raft_consensus.cc:3037] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:59.231689 20662 raft_consensus.cc:3037] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:39:59.232406 20793 raft_consensus.cc:2454] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f999473a364119a4b5902bc646512a in term 1.
I20260923 14:39:59.232487 20662 raft_consensus.cc:2454] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f999473a364119a4b5902bc646512a in term 1.
I20260923 14:39:59.232659 20859 leader_election.cc:302] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f999473a364119a4b5902bc646512a, b4c5a4513d39453c9d6c12e56c3acc1a; no voters:
I20260923 14:39:59.232812 20993 raft_consensus.cc:2781] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:39:59.233026 20993 raft_consensus.cc:686] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 1 LEADER]: Becoming Leader. State: Replica: 52f999473a364119a4b5902bc646512a, State: Running, Role: LEADER
I20260923 14:39:59.233268 20993 consensus_queue.cc:237] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 } } peers { permanent_uuid: "52f999473a364119a4b5902bc646512a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 40811 } } peers { permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 } }
I20260923 14:39:59.235832 20519 catalog_manager.cc:5495] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f999473a364119a4b5902bc646512a (127.2.218.3). New cstate: current_term: 1 leader_uuid: "52f999473a364119a4b5902bc646512a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f999473a364119a4b5902bc646512a" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 40811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 } health_report { overall_health: UNKNOWN } } }
I20260923 14:39:59.262611 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.259929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.2.218.3:40811)
0923 14:39:59.260025 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 14:39:59.260028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:39:59.260039 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:39:59.260156 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:39:59.260158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:39:59.260195 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:39:59.260263 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:39:59.260458 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.260844 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.261560 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:39:59.261723 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:39:59.261783 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:39:59.261990 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:39:59.262293 (+ 303us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:39:59.262362 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 14:39:59.262423 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 14:39:59.262455 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:39:59.266830 20793 raft_consensus.cc:1261] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a [term 1 FOLLOWER]: Refusing update from remote peer 52f999473a364119a4b5902bc646512a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:39:59.266894 20662 raft_consensus.cc:1261] T 0032320490a14a41af21d08c0712bb59 P 2145556be0924062806249e81a671b9f [term 1 FOLLOWER]: Refusing update from remote peer 52f999473a364119a4b5902bc646512a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:39:59.267450 20997 consensus_queue.cc:1035] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2145556be0924062806249e81a671b9f" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 35747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:39:59.267588 20993 consensus_queue.cc:1035] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c5a4513d39453c9d6c12e56c3acc1a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:39:59.274094 21000 mvcc.cc:205] Tried to move back new op lower bound from 7332554339391750144 to 7332554339261652992. Current Snapshot: MvccSnapshot[applied={T|T < 7332554339391750144}]
I20260923 14:39:59.281148 20518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:39:59.281280 20518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:39:59.283228 20518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d41818d2b1d4b2c89bc59b2dbe28390: Sending DeleteTablet for 3 replicas of tablet 0032320490a14a41af21d08c0712bb59
I20260923 14:39:59.283854 20642 tablet_service.cc:1514] Processing DeleteTablet for tablet 0032320490a14a41af21d08c0712bb59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:39:59 UTC) from {username='slave'} at 127.0.0.1:36356
I20260923 14:39:59.283879 20773 tablet_service.cc:1514] Processing DeleteTablet for tablet 0032320490a14a41af21d08c0712bb59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:39:59 UTC) from {username='slave'} at 127.0.0.1:36706
I20260923 14:39:59.284283 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 20580
I20260923 14:39:59.284322 20904 tablet_service.cc:1514] Processing DeleteTablet for tablet 0032320490a14a41af21d08c0712bb59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:39:59 UTC) from {username='slave'} at 127.0.0.1:52688
I20260923 14:39:59.285247 21011 tablet_replica.cc:330] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: stopping tablet replica
I20260923 14:39:59.285429 21010 tablet_replica.cc:330] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: stopping tablet replica
I20260923 14:39:59.285595 21011 raft_consensus.cc:2229] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:39:59.285656 21010 raft_consensus.cc:2229] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:39:59.285883 21010 raft_consensus.cc:2258] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:39:59.285982 21011 raft_consensus.cc:2258] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:39:59.287168 21010 ts_tablet_manager.cc:1905] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:39:59.287184 21011 ts_tablet_manager.cc:1905] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:39:59.289199 21011 ts_tablet_manager.cc:1918] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:39:59.289294 21011 log.cc:1196] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/0032320490a14a41af21d08c0712bb59
I20260923 14:39:59.289433 21010 ts_tablet_manager.cc:1918] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:39:59.289513 21010 log.cc:1196] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/0032320490a14a41af21d08c0712bb59
I20260923 14:39:59.289564 21011 ts_tablet_manager.cc:1939] T 0032320490a14a41af21d08c0712bb59 P 52f999473a364119a4b5902bc646512a: Deleting consensus metadata
I20260923 14:39:59.289793 21010 ts_tablet_manager.cc:1939] T 0032320490a14a41af21d08c0712bb59 P b4c5a4513d39453c9d6c12e56c3acc1a: Deleting consensus metadata
I20260923 14:39:59.290441 20506 catalog_manager.cc:4841] TS 52f999473a364119a4b5902bc646512a (127.2.218.3:40811): tablet 0032320490a14a41af21d08c0712bb59 (table test-table [id=3802249d82ac472b89abc575f8efbcae]) successfully deleted
I20260923 14:39:59.290673 20506 catalog_manager.cc:4841] TS b4c5a4513d39453c9d6c12e56c3acc1a (127.2.218.2:40529): tablet 0032320490a14a41af21d08c0712bb59 (table test-table [id=3802249d82ac472b89abc575f8efbcae]) successfully deleted
W20260923 14:39:59.293823 20505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:35747 (error 108)
I20260923 14:39:59.294031 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 20711
I20260923 14:39:59.295308 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:39:59.295073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:35747 (local address 127.0.0.1:36364)
0923 14:39:59.295186 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 14:39:59.295245 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 14:39:59.298592 20505 catalog_manager.cc:4584] TS 2145556be0924062806249e81a671b9f (127.2.218.1:35747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0032320490a14a41af21d08c0712bb59: Network error: Client connection negotiation failed: client connection to 127.2.218.1:35747: connect: Connection refused (error 111)
I20260923 14:39:59.299376 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 20842
I20260923 14:39:59.304414 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 20489
2026-09-23T14:39:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (890 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:39:59 dist-test-slave-zl47 krb5kdc[21015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:39:59 dist-test-slave-zl47 krb5kdc[21015](info): set up 2 sockets
Sep 23 14:39:59 dist-test-slave-zl47 krb5kdc[21015](info): commencing operation
krb5kdc: starting...
W20260923 14:40:00.202327 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:00 dist-test-slave-zl47 krb5kdc[21015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:40:00.300657 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:38513
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:00.393211 21028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:00.393426 21028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:00.393494 21028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:00.397068 21028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:00.397146 21028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:00.397184 21028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:00.397243 21028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:00.397281 21028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:00.401721 21028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:00.402575 21028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:00.403334 21028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:00.410020 21033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:00.410012 21034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:00.410089 21036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:00.410184 21028 server_base.cc:1034] running on GCE node
I20260923 14:40:00.410712 21028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:00.411513 21028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:00.412684 21028 hybrid_clock.cc:648] HybridClock initialized: now 1790174400412666 us; error 34 us; skew 500 ppm
Sep 23 14:40:00 dist-test-slave-zl47 krb5kdc[21015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:00.420135 21028 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:40:00.420825 21028 webserver.cc:460] Webserver started at http://127.2.218.62:36993/ using document root <none> and password file <none>
I20260923 14:40:00.421275 21028 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:00.421352 21028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:00.421546 21028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:00.423147 21028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "70b01fce84ef4a8896e293491dddd61d"
format_stamp: "Formatted at 2026-09-23 14:40:00 on dist-test-slave-zl47"
server_key: "24c09a9a602911070d5ce7519447c2b9"
server_key_iv: "49b80f96e5c0d8d3fd99f763d480960b"
server_key_version: "encryptionkey@0"
I20260923 14:40:00.423617 21028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "70b01fce84ef4a8896e293491dddd61d"
format_stamp: "Formatted at 2026-09-23 14:40:00 on dist-test-slave-zl47"
server_key: "24c09a9a602911070d5ce7519447c2b9"
server_key_iv: "49b80f96e5c0d8d3fd99f763d480960b"
server_key_version: "encryptionkey@0"
I20260923 14:40:00.426600 21028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:40:00.428565 21043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:00.429492 21028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:00.429608 21028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "70b01fce84ef4a8896e293491dddd61d"
format_stamp: "Formatted at 2026-09-23 14:40:00 on dist-test-slave-zl47"
server_key: "24c09a9a602911070d5ce7519447c2b9"
server_key_iv: "49b80f96e5c0d8d3fd99f763d480960b"
server_key_version: "encryptionkey@0"
I20260923 14:40:00.429699 21028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:00.443662 21028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:00.445825 21028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:00.445988 21028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:00.452024 21028 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:38513
I20260923 14:40:00.452046 21095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:38513 every 8 connection(s)
I20260923 14:40:00.452734 21028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:00.454998 21096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:00.455415 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 21028
I20260923 14:40:00.455533 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:00.455826 2920 external_mini_cluster.cc:1428] Setting key 0eeab0b04a033b2d2776cd7bbe6de893
I20260923 14:40:00.460788 21096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d: Bootstrap starting.
I20260923 14:40:00.462875 21096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:00.463532 21096 log.cc:824] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d: Log is configured to *not* fsync() on all Append() calls
Sep 23 14:40:00 dist-test-slave-zl47 krb5kdc[21015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:00.465046 21096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d: No bootstrap required, opened a new log
I20260923 14:40:00.467918 21096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b01fce84ef4a8896e293491dddd61d" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38513 } }
I20260923 14:40:00.468109 21096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:00.468176 21096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b01fce84ef4a8896e293491dddd61d, State: Initialized, Role: FOLLOWER
I20260923 14:40:00.468201 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:00.457037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.2.218.62:38513)
0923 14:40:00.457411 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:00.457425 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:00.457463 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:40:00.458203 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:00.458222 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:00.458585 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:00.458946 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:00.459544 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:00.460777 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:00.461544 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:00.462149 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:00.462281 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:00.464427 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:00.464431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:00.464441 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:00.464461 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:00.465842 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:00.466295 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:00.466299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:00.466302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:00.466402 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:00.466653 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:00.466656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:00.466657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:00.466957 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:00.467140 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:00.467337 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 14:40:00.467571 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":94,"threads_started":1}
I20260923 14:40:00.468600 21096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b01fce84ef4a8896e293491dddd61d" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38513 } }
I20260923 14:40:00.468761 21096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:00.468819 21096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:00.468899 21096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:00.469627 21096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b01fce84ef4a8896e293491dddd61d" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38513 } }
I20260923 14:40:00.469928 21096 leader_election.cc:302] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70b01fce84ef4a8896e293491dddd61d; no voters:
I20260923 14:40:00.470243 21096 leader_election.cc:288] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:00.470341 21101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:00.470531 21101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [term 1 LEADER]: Becoming Leader. State: Replica: 70b01fce84ef4a8896e293491dddd61d, State: Running, Role: LEADER
I20260923 14:40:00.470813 21101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b01fce84ef4a8896e293491dddd61d" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38513 } }
I20260923 14:40:00.471261 21096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:00.472293 21102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70b01fce84ef4a8896e293491dddd61d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b01fce84ef4a8896e293491dddd61d" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38513 } } }
I20260923 14:40:00.472435 21102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:00.472661 21103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70b01fce84ef4a8896e293491dddd61d. Latest consensus state: current_term: 1 leader_uuid: "70b01fce84ef4a8896e293491dddd61d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b01fce84ef4a8896e293491dddd61d" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 38513 } } }
I20260923 14:40:00.472731 21103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:00.472775 21110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:00.475354 21110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:00.479849 21110 catalog_manager.cc:1347] Generated new cluster ID: e98b6e9db617482bae891ae7bc4f8e63
I20260923 14:40:00.479914 21110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:00.506855 21110 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:00.507774 21110 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:00.521622 21110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d: Generated new TSK 0
I20260923 14:40:00.522208 21110 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:40:00.554780 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38513
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:00.646791 21122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:00.646966 21122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:00.646999 21122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:00.650379 21122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:00.650419 21122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:00.650473 21122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:40:00.656000 21122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:00.656989 21122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:00.658010 21122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:00.666196 21130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:00.666193 21127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:00.666266 21122 server_base.cc:1034] running on GCE node
W20260923 14:40:00.666194 21128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:00.666741 21122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:00.667249 21122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:00.668423 21122 hybrid_clock.cc:648] HybridClock initialized: now 1790174400668410 us; error 26 us; skew 500 ppm
Sep 23 14:40:00 dist-test-slave-zl47 krb5kdc[21015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:00.670746 21122 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:40:00.671571 21122 webserver.cc:460] Webserver started at http://127.2.218.1:46335/ using document root <none> and password file <none>
I20260923 14:40:00.672040 21122 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:00.672119 21122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:00.672353 21122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:00.674067 21122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "e55f79bc697148c09eba5d957ca9fc2b"
format_stamp: "Formatted at 2026-09-23 14:40:00 on dist-test-slave-zl47"
server_key: "6b8106a4a6fdb5a232593530f567ac6b"
server_key_iv: "73fbfc21341f66d9141cadc1efe4d86e"
server_key_version: "encryptionkey@0"
I20260923 14:40:00.674580 21122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "e55f79bc697148c09eba5d957ca9fc2b"
format_stamp: "Formatted at 2026-09-23 14:40:00 on dist-test-slave-zl47"
server_key: "6b8106a4a6fdb5a232593530f567ac6b"
server_key_iv: "73fbfc21341f66d9141cadc1efe4d86e"
server_key_version: "encryptionkey@0"
I20260923 14:40:00.677413 21122 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:40:00.679445 21137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:00.680469 21122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:40:00.680567 21122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "e55f79bc697148c09eba5d957ca9fc2b"
format_stamp: "Formatted at 2026-09-23 14:40:00 on dist-test-slave-zl47"
server_key: "6b8106a4a6fdb5a232593530f567ac6b"
server_key_iv: "73fbfc21341f66d9141cadc1efe4d86e"
server_key_version: "encryptionkey@0"
I20260923 14:40:00.680657 21122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:00.693086 21122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:00.695887 21122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:00.696048 21122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:00.696506 21122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:00.697250 21122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:00.697309 21122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:00.697351 21122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:00.697378 21122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:00.705956 21122 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:40409
I20260923 14:40:00.705987 21250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:40409 every 8 connection(s)
I20260923 14:40:00.706693 21122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:40:00.709693 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 21122
I20260923 14:40:00.709805 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:40:00.710124 2920 external_mini_cluster.cc:1428] Setting key 41ab2c8e8cd79f8818731f1adf4d8641
Sep 23 14:40:00 dist-test-slave-zl47 krb5kdc[21015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
I20260923 14:40:00.719309 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:00.708199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:54957 (local address 127.2.218.62:38513)
0923 14:40:00.708366 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:00.708392 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:00.709239 (+ 847us) server_negotiation.cc:407] Connection header received
0923 14:40:00.710133 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:00.710139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:00.710193 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:00.710275 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:00.711300 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:00.711969 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:00.713258 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:00.713468 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:00.713548 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:00.716221 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:00.716225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:00.716228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:00.716258 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:00.717560 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:00.718002 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:00.718006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:00.718008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:00.718078 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:00.718408 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:00.718411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:00.718413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:00.718566 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:00.718660 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:00.719062 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 14:40:00.719165 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 14:40:00.719918 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:00.708520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38513 (local address 127.2.218.1:54957)
0923 14:40:00.709109 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:00.709145 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:00.709947 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:00.710407 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:00.710413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:00.710771 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:00.711149 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:00.711181 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:00.712104 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:00.712106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:00.713111 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:00.713124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:00.713623 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:00.713628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:00.713846 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:00.714390 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:00.714415 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:00.716063 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:00.717669 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:00.717674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:00.717685 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:00.717910 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:00.718199 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:00.718201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:00.718203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:00.718320 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:00.718659 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:00.718664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:00.718859 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 14:40:00.719013 (+ 154us) client_negotiation.cc:240] Negotiation successful
0923 14:40:00.719261 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":99,"threads_started":1}
I20260923 14:40:00.721122 21251 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38513
I20260923 14:40:00.721441 21251 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
I20260923 14:40:00.722014 21251 heartbeater.cc:511] Master 127.2.218.62:38513 requested a full tablet report, sending...
I20260923 14:40:00.723515 21060 ts_manager.cc:194] Registered new tserver with Master: e55f79bc697148c09eba5d957ca9fc2b (127.2.218.1:40409)
I20260923 14:40:00.724517 21060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:54957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:40:00.734864 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38513
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:00.826886 21256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:00.827073 21256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:00.827111 21256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:00.830559 21256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:00.830611 21256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:00.830662 21256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:40:00.835012 21256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:00.835754 21256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:00.836532 21256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:00.844123 21264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:00.844123 21261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:00.844190 21262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:00.844343 21256 server_base.cc:1034] running on GCE node
I20260923 14:40:00.844710 21256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:00.845212 21256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:00.846388 21256 hybrid_clock.cc:648] HybridClock initialized: now 1790174400846374 us; error 45 us; skew 500 ppm
Sep 23 14:40:00 dist-test-slave-zl47 krb5kdc[21015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:00.848628 21256 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:40:00.849303 21256 webserver.cc:460] Webserver started at http://127.2.218.2:35813/ using document root <none> and password file <none>
I20260923 14:40:00.849752 21256 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:00.849836 21256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:00.850041 21256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:00.851634 21256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a"
format_stamp: "Formatted at 2026-09-23 14:40:00 on dist-test-slave-zl47"
server_key: "086ce151b8ccdd4851388c3544ea312b"
server_key_iv: "2b77fffeac2b6faf9911e28b66d028ee"
server_key_version: "encryptionkey@0"
I20260923 14:40:00.852128 21256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a"
format_stamp: "Formatted at 2026-09-23 14:40:00 on dist-test-slave-zl47"
server_key: "086ce151b8ccdd4851388c3544ea312b"
server_key_iv: "2b77fffeac2b6faf9911e28b66d028ee"
server_key_version: "encryptionkey@0"
I20260923 14:40:00.855137 21256 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:40:00.857143 21271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:00.858261 21256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:40:00.858369 21256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a"
format_stamp: "Formatted at 2026-09-23 14:40:00 on dist-test-slave-zl47"
server_key: "086ce151b8ccdd4851388c3544ea312b"
server_key_iv: "2b77fffeac2b6faf9911e28b66d028ee"
server_key_version: "encryptionkey@0"
I20260923 14:40:00.858471 21256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:00.911876 21256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:00.914119 21256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:00.914297 21256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:00.914804 21256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:00.915457 21256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:00.915519 21256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:00.915575 21256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:00.915606 21256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:00.923105 21256 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:35005
I20260923 14:40:00.923137 21384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:35005 every 8 connection(s)
I20260923 14:40:00.923787 21256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:40:00.929584 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 21256
I20260923 14:40:00.929703 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:40:00.929971 2920 external_mini_cluster.cc:1428] Setting key 2246cb7b92e6f7627b12a61f6ec01b01
Sep 23 14:40:00 dist-test-slave-zl47 krb5kdc[21015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174400, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:00.935870 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:00.925323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:53679 (local address 127.2.218.62:38513)
0923 14:40:00.925456 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:00.925459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:00.926081 (+ 622us) server_negotiation.cc:407] Connection header received
0923 14:40:00.926854 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:00.926857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:00.926921 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:00.927007 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:00.927952 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:00.928479 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:00.929463 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:00.929701 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:00.929821 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:00.932426 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:00.932428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:00.932431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:00.932448 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:00.933927 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:00.934495 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:00.934497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:00.934498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:00.934563 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:00.934922 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:00.934924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:00.934925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:00.935064 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:00.935128 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:00.935683 (+ 555us) server_negotiation.cc:299] Negotiation successful
0923 14:40:00.935766 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 14:40:00.936818 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:00.925595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38513 (local address 127.2.218.2:53679)
0923 14:40:00.925953 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:00.925986 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:00.926673 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:00.927154 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:00.927159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:00.927443 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:00.927803 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:00.927814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:00.928591 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:00.928594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:00.929333 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:00.929342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:00.929959 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:00.929962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:00.930195 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:00.930715 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:00.930731 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:00.932246 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:00.934027 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:00.934031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:00.934040 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:00.934396 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:00.934643 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:00.934648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:00.934652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:00.934815 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:00.935157 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:00.935165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:00.935450 (+ 285us) client_negotiation.cc:769] Sending connection context
0923 14:40:00.935634 (+ 184us) client_negotiation.cc:240] Negotiation successful
0923 14:40:00.935926 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":92,"threads_started":1}
I20260923 14:40:00.937755 21385 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38513
I20260923 14:40:00.937951 21385 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:00.938437 21385 heartbeater.cc:511] Master 127.2.218.62:38513 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
I20260923 14:40:00.939347 21060 ts_manager.cc:194] Registered new tserver with Master: 8f5d59be8b5f4fbdb6ac290e6d12903a (127.2.218.2:35005)
I20260923 14:40:00.939837 21060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:53679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:40:00.953714 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38513
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:01.047789 21390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:01.048022 21390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:01.048089 21390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:01.051661 21390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:01.051738 21390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:01.051822 21390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:40:01.056339 21390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:01.057144 21390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:01.057943 21390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:01.065759 21398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:01.065819 21396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:01.065759 21395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:01.065932 21390 server_base.cc:1034] running on GCE node
I20260923 14:40:01.066359 21390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:01.066869 21390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:01.068044 21390 hybrid_clock.cc:648] HybridClock initialized: now 1790174401068032 us; error 29 us; skew 500 ppm
Sep 23 14:40:01 dist-test-slave-zl47 krb5kdc[21015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174401, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:01.070443 21390 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:40:01.071205 21390 webserver.cc:460] Webserver started at http://127.2.218.3:42223/ using document root <none> and password file <none>
I20260923 14:40:01.071740 21390 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:01.071821 21390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:01.072023 21390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:01.073861 21390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "b08be1650e56436e8a706f3af47e1140"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "a11face8d603a8310e31383b313afbca"
server_key_iv: "87fcde3903d98d4449d795ab7b338128"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.074388 21390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "b08be1650e56436e8a706f3af47e1140"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "a11face8d603a8310e31383b313afbca"
server_key_iv: "87fcde3903d98d4449d795ab7b338128"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.077443 21390 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260923 14:40:01.079569 21405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.080600 21390 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:40:01.080718 21390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "b08be1650e56436e8a706f3af47e1140"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "a11face8d603a8310e31383b313afbca"
server_key_iv: "87fcde3903d98d4449d795ab7b338128"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.080806 21390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:01.098627 21390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:01.100806 21390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:01.100970 21390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:01.101418 21390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:01.102043 21390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:01.102125 21390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.102200 21390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:01.102249 21390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.109543 21390 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:43199
I20260923 14:40:01.109566 21518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:43199 every 8 connection(s)
I20260923 14:40:01.110203 21390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
Sep 23 14:40:01 dist-test-slave-zl47 krb5kdc[21015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174401, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:01.118404 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 21390
I20260923 14:40:01.118523 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:40:01.118813 2920 external_mini_cluster.cc:1428] Setting key 8b3586c2fc29821b241b12111b10d1e0
I20260923 14:40:01.121974 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.111677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:42249 (local address 127.2.218.62:38513)
0923 14:40:01.111822 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.111826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.112573 (+ 747us) server_negotiation.cc:407] Connection header received
0923 14:40:01.113241 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.113243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.113282 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.113345 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:01.114336 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.114788 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.115811 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.116030 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.116141 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.118724 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:01.118727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:01.118729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:01.118750 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:01.120077 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:01.120561 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:01.120563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:01.120564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:01.120629 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:01.120935 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:01.120939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:01.120940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:01.121109 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:01.121204 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.121700 (+ 496us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.121811 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 14:40:01.122613 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.111872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:38513 (local address 127.2.218.3:42249)
0923 14:40:01.112451 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:01.112484 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:01.113087 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:01.113481 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:01.113486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:01.113758 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:01.114222 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.114236 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.114894 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.114896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.115683 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.115693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.116170 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.116174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.116390 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.116960 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:01.116979 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:01.118554 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:01.120160 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:01.120166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:01.120177 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:01.120471 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:01.120718 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:01.120722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:01.120723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:01.120847 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:01.121205 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:01.121211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:01.121459 (+ 248us) client_negotiation.cc:769] Sending connection context
0923 14:40:01.121661 (+ 202us) client_negotiation.cc:240] Negotiation successful
0923 14:40:01.121922 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":94,"threads_started":1}
I20260923 14:40:01.123517 21519 heartbeater.cc:347] Connected to a master server at 127.2.218.62:38513
I20260923 14:40:01.123665 21519 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:01.123991 21519 heartbeater.cc:511] Master 127.2.218.62:38513 requested a full tablet report, sending...
I20260923 14:40:01.124857 21060 ts_manager.cc:194] Registered new tserver with Master: b08be1650e56436e8a706f3af47e1140 (127.2.218.3:43199)
I20260923 14:40:01.125356 21060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:42249
I20260923 14:40:01.132079 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:40:01.139726 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.133645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.2.218.62:38513)
0923 14:40:01.133767 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.133770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.133935 (+ 165us) server_negotiation.cc:407] Connection header received
0923 14:40:01.133999 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.134001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.134028 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.134080 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:01.134633 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.135031 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.135713 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.135896 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.135976 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.137125 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:01.137128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:01.137129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:01.137148 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:01.138310 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:01.138699 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:01.138701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:01.138702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:01.138767 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:01.139008 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:01.139010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:01.139011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:01.139163 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:01.139270 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.139473 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.139568 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:40:01.142057 21060 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:40:01.144543 21060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:40:01.156797 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.151950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:40409 (local address 127.0.0.1:44824)
0923 14:40:01.152411 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:01.152425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:01.152505 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:01.153418 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:01.153421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:01.153435 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:01.153509 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.153514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.154579 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.154585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.155234 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.155242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.156215 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.156218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.156381 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.156448 (+ 67us) client_negotiation.cc:769] Sending connection context
0923 14:40:01.156542 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 14:40:01.156658 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":37,"thread_start_us":94,"threads_started":1}
I20260923 14:40:01.156798 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.151559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:35005 (local address 127.0.0.1:44036)
0923 14:40:01.152125 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:01.152141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:01.152226 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:01.152753 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:01.152756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:01.152788 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:01.152904 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.152910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.154056 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.154059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.155216 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.155226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.156215 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.156218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.156366 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.156449 (+ 83us) client_negotiation.cc:769] Sending connection context
0923 14:40:01.156542 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 14:40:01.156659 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":169,"spinlock_wait_cycles":2304,"thread_start_us":69,"threads_started":1}
I20260923 14:40:01.156831 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.151929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.2.218.2:35005)
0923 14:40:01.152297 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.152306 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.152322 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:40:01.152387 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.152391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.152513 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.152638 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:01.153034 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.153924 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.155342 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.156099 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.156207 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.156332 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.156594 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.156688 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":96,"threads_started":1}
I20260923 14:40:01.157335 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.152348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.2.218.1:40409)
0923 14:40:01.152889 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.152898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.152918 (+ 20us) server_negotiation.cc:407] Connection header received
0923 14:40:01.152984 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.152990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.153142 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.153265 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:01.153630 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.154456 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.155400 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.156099 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.156207 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.156332 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.157084 (+ 752us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.157184 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":115,"threads_started":1}
I20260923 14:40:01.157823 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.151374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:43199 (local address 127.0.0.1:56644)
0923 14:40:01.151991 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:01.152034 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:01.152140 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:01.152740 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:01.152747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:01.152767 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:01.152890 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.152897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.153927 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.153930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.155219 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.155226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.157413 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.157416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.157576 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.157595 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 14:40:01.157653 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 14:40:01.157710 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":92,"spinlock_wait_cycles":1920,"thread_start_us":68,"threads_started":1}
I20260923 14:40:01.158021 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.151467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.2.218.3:43199)
0923 14:40:01.152010 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.152026 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.152255 (+ 229us) server_negotiation.cc:407] Connection header received
0923 14:40:01.152314 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.152319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.152472 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.152598 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:01.153004 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.153807 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.156392 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.157290 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.157497 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.157646 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.157690 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.157761 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":97,"threads_started":1}
I20260923 14:40:01.159561 21319 tablet_service.cc:1467] Processing CreateTablet for tablet 8760045668ea42a3bab15c1db1516e01 (DEFAULT_TABLE table=test-table [id=ae60b1c9db064b67b3af599d03489196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:01.159529 21185 tablet_service.cc:1467] Processing CreateTablet for tablet 8760045668ea42a3bab15c1db1516e01 (DEFAULT_TABLE table=test-table [id=ae60b1c9db064b67b3af599d03489196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:01.160320 21319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8760045668ea42a3bab15c1db1516e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:01.160321 21185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8760045668ea42a3bab15c1db1516e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:01.164149 21453 tablet_service.cc:1467] Processing CreateTablet for tablet 8760045668ea42a3bab15c1db1516e01 (DEFAULT_TABLE table=test-table [id=ae60b1c9db064b67b3af599d03489196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:01.164772 21453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8760045668ea42a3bab15c1db1516e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:01.166304 21536 tablet_bootstrap.cc:492] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b: Bootstrap starting.
I20260923 14:40:01.168416 21536 tablet_bootstrap.cc:654] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:01.169576 21536 log.cc:824] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:01.169749 21537 tablet_bootstrap.cc:492] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: Bootstrap starting.
I20260923 14:40:01.170480 21539 tablet_bootstrap.cc:492] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: Bootstrap starting.
I20260923 14:40:01.171396 21536 tablet_bootstrap.cc:492] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b: No bootstrap required, opened a new log
I20260923 14:40:01.171620 21536 ts_tablet_manager.cc:1397] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:40:01.171903 21537 tablet_bootstrap.cc:654] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:01.172742 21539 tablet_bootstrap.cc:654] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:01.173422 21537 log.cc:824] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:01.173445 21539 log.cc:824] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:01.175320 21537 tablet_bootstrap.cc:492] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: No bootstrap required, opened a new log
I20260923 14:40:01.175319 21539 tablet_bootstrap.cc:492] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: No bootstrap required, opened a new log
I20260923 14:40:01.175274 21536 raft_consensus.cc:348] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08be1650e56436e8a706f3af47e1140" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43199 } } peers { permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 } } peers { permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 } }
I20260923 14:40:01.175493 21539 ts_tablet_manager.cc:1397] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:40:01.175493 21537 ts_tablet_manager.cc:1397] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260923 14:40:01.175544 21536 raft_consensus.cc:374] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:01.175616 21536 raft_consensus.cc:729] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55f79bc697148c09eba5d957ca9fc2b, State: Initialized, Role: FOLLOWER
I20260923 14:40:01.176055 21536 consensus_queue.cc:260] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08be1650e56436e8a706f3af47e1140" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43199 } } peers { permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 } } peers { permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 } }
I20260923 14:40:01.176959 21536 ts_tablet_manager.cc:1428] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:40:01.177266 21251 heartbeater.cc:503] Master 127.2.218.62:38513 was elected leader, sending a full tablet report...
I20260923 14:40:01.179147 21537 raft_consensus.cc:348] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08be1650e56436e8a706f3af47e1140" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43199 } } peers { permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 } } peers { permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 } }
I20260923 14:40:01.179147 21539 raft_consensus.cc:348] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08be1650e56436e8a706f3af47e1140" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43199 } } peers { permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 } } peers { permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 } }
I20260923 14:40:01.179425 21539 raft_consensus.cc:374] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:01.179417 21537 raft_consensus.cc:374] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:01.179474 21539 raft_consensus.cc:729] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5d59be8b5f4fbdb6ac290e6d12903a, State: Initialized, Role: FOLLOWER
I20260923 14:40:01.179481 21537 raft_consensus.cc:729] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08be1650e56436e8a706f3af47e1140, State: Initialized, Role: FOLLOWER
I20260923 14:40:01.179841 21539 consensus_queue.cc:260] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08be1650e56436e8a706f3af47e1140" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43199 } } peers { permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 } } peers { permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 } }
I20260923 14:40:01.179934 21537 consensus_queue.cc:260] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08be1650e56436e8a706f3af47e1140" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43199 } } peers { permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 } } peers { permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 } }
I20260923 14:40:01.180560 21519 heartbeater.cc:503] Master 127.2.218.62:38513 was elected leader, sending a full tablet report...
I20260923 14:40:01.180645 21539 ts_tablet_manager.cc:1428] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260923 14:40:01.180653 21537 ts_tablet_manager.cc:1428] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260923 14:40:01.180848 21385 heartbeater.cc:503] Master 127.2.218.62:38513 was elected leader, sending a full tablet report...
I20260923 14:40:01.199242 21544 raft_consensus.cc:482] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:40:01.199359 21544 raft_consensus.cc:504] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08be1650e56436e8a706f3af47e1140" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43199 } } peers { permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 } } peers { permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 } }
I20260923 14:40:01.200160 21544 leader_election.cc:288] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f5d59be8b5f4fbdb6ac290e6d12903a (127.2.218.2:35005), e55f79bc697148c09eba5d957ca9fc2b (127.2.218.1:40409)
I20260923 14:40:01.203300 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.200426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:45545 (local address 127.2.218.2:35005)
0923 14:40:01.200532 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.200536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.200547 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:40:01.200582 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.200585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.200619 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.200688 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:01.200939 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.201444 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.202425 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.202791 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.202858 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.202898 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.203117 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.203196 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:40:01.203334 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.200409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:53185 (local address 127.2.218.1:40409)
0923 14:40:01.200513 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.200516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.200725 (+ 209us) server_negotiation.cc:407] Connection header received
0923 14:40:01.200809 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.200813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.200849 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.200916 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:01.201149 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.201580 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.202508 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.202817 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.202949 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.202998 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.203160 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.203248 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:40:01.203317 21521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.200267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:35005 (local address 127.2.218.3:45545)
0923 14:40:01.200366 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:01.200379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:01.200443 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:01.200721 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:01.200724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:01.200743 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:01.200837 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.200843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.201540 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.201543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.202308 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.202316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.202910 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.202914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.203001 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.203023 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:40:01.203082 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 14:40:01.203152 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 14:40:01.203330 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.200395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:40409 (local address 127.2.218.3:53185)
0923 14:40:01.200651 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:01.200662 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:01.200733 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:01.200939 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:01.200941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:01.200955 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:01.201049 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.201054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.201671 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.201673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.202405 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.202415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.202968 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.202970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.203036 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.203050 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 14:40:01.203130 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 14:40:01.203198 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260923 14:40:01.203855 21339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8760045668ea42a3bab15c1db1516e01" candidate_uuid: "b08be1650e56436e8a706f3af47e1140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" is_pre_election: true
I20260923 14:40:01.204062 21339 raft_consensus.cc:2454] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b08be1650e56436e8a706f3af47e1140 in term 0.
I20260923 14:40:01.204030 21205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8760045668ea42a3bab15c1db1516e01" candidate_uuid: "b08be1650e56436e8a706f3af47e1140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e55f79bc697148c09eba5d957ca9fc2b" is_pre_election: true
I20260923 14:40:01.204254 21205 raft_consensus.cc:2454] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b08be1650e56436e8a706f3af47e1140 in term 0.
I20260923 14:40:01.204444 21407 leader_election.cc:302] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5d59be8b5f4fbdb6ac290e6d12903a, b08be1650e56436e8a706f3af47e1140; no voters:
I20260923 14:40:01.204631 21544 raft_consensus.cc:2781] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:40:01.204697 21544 raft_consensus.cc:482] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:40:01.204733 21544 raft_consensus.cc:3037] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:01.205435 21544 raft_consensus.cc:504] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08be1650e56436e8a706f3af47e1140" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43199 } } peers { permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 } } peers { permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 } }
I20260923 14:40:01.205788 21544 leader_election.cc:288] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [CANDIDATE]: Term 1 election: Requested vote from peers 8f5d59be8b5f4fbdb6ac290e6d12903a (127.2.218.2:35005), e55f79bc697148c09eba5d957ca9fc2b (127.2.218.1:40409)
I20260923 14:40:01.206032 21339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8760045668ea42a3bab15c1db1516e01" candidate_uuid: "b08be1650e56436e8a706f3af47e1140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a"
I20260923 14:40:01.206146 21339 raft_consensus.cc:3037] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:01.206172 21205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8760045668ea42a3bab15c1db1516e01" candidate_uuid: "b08be1650e56436e8a706f3af47e1140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e55f79bc697148c09eba5d957ca9fc2b"
I20260923 14:40:01.206284 21205 raft_consensus.cc:3037] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:01.206972 21339 raft_consensus.cc:2454] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b08be1650e56436e8a706f3af47e1140 in term 1.
I20260923 14:40:01.207115 21205 raft_consensus.cc:2454] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b08be1650e56436e8a706f3af47e1140 in term 1.
I20260923 14:40:01.207245 21407 leader_election.cc:302] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f5d59be8b5f4fbdb6ac290e6d12903a, b08be1650e56436e8a706f3af47e1140; no voters:
I20260923 14:40:01.207403 21544 raft_consensus.cc:2781] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 1 FOLLOWER]: Leader election won for term 1
W20260923 14:40:01.207625 21252 tablet.cc:2369] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:40:01.207615 21544 raft_consensus.cc:686] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 1 LEADER]: Becoming Leader. State: Replica: b08be1650e56436e8a706f3af47e1140, State: Running, Role: LEADER
I20260923 14:40:01.207854 21544 consensus_queue.cc:237] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08be1650e56436e8a706f3af47e1140" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43199 } } peers { permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 } } peers { permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 } }
I20260923 14:40:01.210235 21059 catalog_manager.cc:5495] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 reported cstate change: term changed from 0 to 1, leader changed from <none> to b08be1650e56436e8a706f3af47e1140 (127.2.218.3). New cstate: current_term: 1 leader_uuid: "b08be1650e56436e8a706f3af47e1140" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08be1650e56436e8a706f3af47e1140" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 43199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 } health_report { overall_health: UNKNOWN } } }
I20260923 14:40:01.233894 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.230628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.2.218.3:43199)
0923 14:40:01.230721 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.230724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.230791 (+ 67us) server_negotiation.cc:407] Connection header received
0923 14:40:01.230906 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.230908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.230945 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.231015 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:01.231209 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.231605 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.232431 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.232609 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.232679 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.233027 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:01.233450 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:01.233561 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.233683 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.233731 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:40:01.238415 21205 raft_consensus.cc:1261] T 8760045668ea42a3bab15c1db1516e01 P e55f79bc697148c09eba5d957ca9fc2b [term 1 FOLLOWER]: Refusing update from remote peer b08be1650e56436e8a706f3af47e1140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:01.238417 21339 raft_consensus.cc:1261] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a [term 1 FOLLOWER]: Refusing update from remote peer b08be1650e56436e8a706f3af47e1140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:01.238972 21546 consensus_queue.cc:1035] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e55f79bc697148c09eba5d957ca9fc2b" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:01.239094 21544 consensus_queue.cc:1035] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f5d59be8b5f4fbdb6ac290e6d12903a" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:01.247792 21550 mvcc.cc:205] Tried to move back new op lower bound from 7332554347466813440 to 7332554347349413888. Current Snapshot: MvccSnapshot[applied={T|T < 7332554347466813440}]
I20260923 14:40:01.252310 21060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:40:01.252502 21060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:40:01.254475 21060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70b01fce84ef4a8896e293491dddd61d: Sending DeleteTablet for 3 replicas of tablet 8760045668ea42a3bab15c1db1516e01
I20260923 14:40:01.255152 21319 tablet_service.cc:1514] Processing DeleteTablet for tablet 8760045668ea42a3bab15c1db1516e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:01 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:44036
I20260923 14:40:01.255236 21453 tablet_service.cc:1514] Processing DeleteTablet for tablet 8760045668ea42a3bab15c1db1516e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:01 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:56644
I20260923 14:40:01.255585 21559 tablet_replica.cc:330] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: stopping tablet replica
I20260923 14:40:01.255606 21185 tablet_service.cc:1514] Processing DeleteTablet for tablet 8760045668ea42a3bab15c1db1516e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:01 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:44824
I20260923 14:40:01.255641 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 21122
I20260923 14:40:01.255784 21560 tablet_replica.cc:330] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: stopping tablet replica
I20260923 14:40:01.255986 21559 raft_consensus.cc:2229] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:40:01.256456 21559 raft_consensus.cc:2258] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:40:01.257608 21560 raft_consensus.cc:2229] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:40:01.257668 21559 ts_tablet_manager.cc:1905] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:40:01.258034 21560 raft_consensus.cc:2258] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:40:01.259311 21560 ts_tablet_manager.cc:1905] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:40:01.260116 21559 ts_tablet_manager.cc:1918] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:40:01.260198 21559 log.cc:1196] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/8760045668ea42a3bab15c1db1516e01
I20260923 14:40:01.260524 21559 ts_tablet_manager.cc:1939] T 8760045668ea42a3bab15c1db1516e01 P 8f5d59be8b5f4fbdb6ac290e6d12903a: Deleting consensus metadata
I20260923 14:40:01.261556 21045 catalog_manager.cc:4841] TS 8f5d59be8b5f4fbdb6ac290e6d12903a (127.2.218.2:35005): tablet 8760045668ea42a3bab15c1db1516e01 (table test-table [id=ae60b1c9db064b67b3af599d03489196]) successfully deleted
I20260923 14:40:01.262732 21560 ts_tablet_manager.cc:1918] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:40:01.262890 21560 log.cc:1196] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/8760045668ea42a3bab15c1db1516e01
I20260923 14:40:01.263253 21560 ts_tablet_manager.cc:1939] T 8760045668ea42a3bab15c1db1516e01 P b08be1650e56436e8a706f3af47e1140: Deleting consensus metadata
I20260923 14:40:01.264263 21046 catalog_manager.cc:4841] TS b08be1650e56436e8a706f3af47e1140 (127.2.218.3:43199): tablet 8760045668ea42a3bab15c1db1516e01 (table test-table [id=ae60b1c9db064b67b3af599d03489196]) successfully deleted
W20260923 14:40:01.264444 21046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:40409 (error 108)
I20260923 14:40:01.264621 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 21256
I20260923 14:40:01.265748 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.265516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:40409 (local address 127.0.0.1:44834)
0923 14:40:01.265639 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:01.265688 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:40409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 14:40:01.265866 21046 catalog_manager.cc:4584] TS e55f79bc697148c09eba5d957ca9fc2b (127.2.218.1:40409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8760045668ea42a3bab15c1db1516e01: Network error: Client connection negotiation failed: client connection to 127.2.218.1:40409: connect: Connection refused (error 111)
I20260923 14:40:01.270880 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 21390
I20260923 14:40:01.275591 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 21028
2026-09-23T14:40:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1971 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T14:40:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:01Z Disabled control of system clock
I20260923 14:40:01.302615 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:39613
--webserver_interface=127.2.218.62
--webserver_port=0
--builtin_ntp_servers=127.2.218.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:40:01.392143 21565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:01.392319 21565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:01.392349 21565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:01.395529 21565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:01.395569 21565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:01.395582 21565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:01.395596 21565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:01.395608 21565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:01.399408 21565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:39613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:01.400058 21565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:01.400995 21565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:01.407624 21570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:01.407606 21571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:01.407686 21573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:01.407761 21565 server_base.cc:1034] running on GCE node
I20260923 14:40:01.408289 21565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:01.409083 21565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:01.410243 21565 hybrid_clock.cc:648] HybridClock initialized: now 1790174401410224 us; error 33 us; skew 500 ppm
I20260923 14:40:01.411669 21565 webserver.cc:460] Webserver started at http://127.2.218.62:32847/ using document root <none> and password file <none>
I20260923 14:40:01.412106 21565 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:01.412184 21565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:01.412350 21565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:01.413949 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "ddf62dec5e4547cca1474d7b78438985"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "308187c2f29674657b2cba154ec0598a"
server_key_iv: "1193f57acdc079d5badcefa6daee6207"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.414486 21565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "ddf62dec5e4547cca1474d7b78438985"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "308187c2f29674657b2cba154ec0598a"
server_key_iv: "1193f57acdc079d5badcefa6daee6207"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.417277 21565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:40:01.419373 21579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.420326 21565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:40:01.420454 21565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "ddf62dec5e4547cca1474d7b78438985"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "308187c2f29674657b2cba154ec0598a"
server_key_iv: "1193f57acdc079d5badcefa6daee6207"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.420542 21565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:01.440274 21565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:01.440770 21565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:01.440936 21565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:01.446866 21565 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:39613
I20260923 14:40:01.446890 21631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:39613 every 8 connection(s)
I20260923 14:40:01.447588 21565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:01.450050 21632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:01.454605 21632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985: Bootstrap starting.
I20260923 14:40:01.456668 21632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:01.456830 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 21565
I20260923 14:40:01.456961 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:01.457203 2920 external_mini_cluster.cc:1428] Setting key 1aabade8d8bc5e4f5106903f64ea73a0
I20260923 14:40:01.457468 21632 log.cc:824] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:01.459326 21632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985: No bootstrap required, opened a new log
I20260923 14:40:01.462802 21632 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf62dec5e4547cca1474d7b78438985" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39613 } }
I20260923 14:40:01.463042 21632 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:01.463127 21632 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf62dec5e4547cca1474d7b78438985, State: Initialized, Role: FOLLOWER
I20260923 14:40:01.463554 21632 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf62dec5e4547cca1474d7b78438985" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39613 } }
I20260923 14:40:01.463707 21632 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:01.463792 21632 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:01.463917 21632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:01.464771 21632 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf62dec5e4547cca1474d7b78438985" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39613 } }
I20260923 14:40:01.465023 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.458242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.2.218.62:39613)
0923 14:40:01.458758 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.458769 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.458795 (+ 26us) server_negotiation.cc:407] Connection header received
0923 14:40:01.459269 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.459282 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.459580 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.459791 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:01.460130 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.460937 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.462137 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.462535 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.462636 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.463477 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:01.463481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:01.463508 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:01.463867 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:01.463930 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:01.463952 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:01.464054 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.464175 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.464399 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":120,"threads_started":1}
I20260923 14:40:01.465106 21632 leader_election.cc:302] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf62dec5e4547cca1474d7b78438985; no voters:
I20260923 14:40:01.465389 21632 leader_election.cc:288] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:01.465473 21637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:01.465641 21637 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [term 1 LEADER]: Becoming Leader. State: Replica: ddf62dec5e4547cca1474d7b78438985, State: Running, Role: LEADER
I20260923 14:40:01.465960 21637 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf62dec5e4547cca1474d7b78438985" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39613 } }
I20260923 14:40:01.466585 21632 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:01.467126 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf62dec5e4547cca1474d7b78438985. Latest consensus state: current_term: 1 leader_uuid: "ddf62dec5e4547cca1474d7b78438985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf62dec5e4547cca1474d7b78438985" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39613 } } }
I20260923 14:40:01.467244 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:01.467603 21642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:01.467625 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf62dec5e4547cca1474d7b78438985" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf62dec5e4547cca1474d7b78438985" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 39613 } } }
I20260923 14:40:01.467806 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:01.470247 21642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:01.474869 21642 catalog_manager.cc:1347] Generated new cluster ID: a0819e5a66414b618d7628b98b338378
I20260923 14:40:01.474932 21642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:01.482933 21642 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:01.483624 21642 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:01.500444 21642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985: Generated new TSK 0
I20260923 14:40:01.500945 21642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:01.505741 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39613
--builtin_ntp_servers=127.2.218.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:40:01.633980 21656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:01.634195 21656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:01.634225 21656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:01.637408 21656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:01.637444 21656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:01.637485 21656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:40:01.641283 21656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:01.641985 21656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:01.643014 21656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:01.650671 21662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:01.650751 21664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:01.650682 21656 server_base.cc:1034] running on GCE node
W20260923 14:40:01.650782 21661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:01.651216 21656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:01.651707 21656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:01.652863 21656 hybrid_clock.cc:648] HybridClock initialized: now 1790174401652840 us; error 36 us; skew 500 ppm
I20260923 14:40:01.654336 21656 webserver.cc:460] Webserver started at http://127.2.218.1:34443/ using document root <none> and password file <none>
I20260923 14:40:01.654788 21656 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:01.654841 21656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:01.655052 21656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:01.656705 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "4738d16ad498426abbda6d21b3f3c6e5"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "b713c43b79db3e6522d4f2ad6393cb04"
server_key_iv: "9f176ed3c3da6de90a65dce43e9f1f83"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.657226 21656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "4738d16ad498426abbda6d21b3f3c6e5"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "b713c43b79db3e6522d4f2ad6393cb04"
server_key_iv: "9f176ed3c3da6de90a65dce43e9f1f83"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.660269 21656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:40:01.662369 21670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.663329 21656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:40:01.663431 21656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "4738d16ad498426abbda6d21b3f3c6e5"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "b713c43b79db3e6522d4f2ad6393cb04"
server_key_iv: "9f176ed3c3da6de90a65dce43e9f1f83"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.663514 21656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:01.673501 21656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:01.673978 21656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:01.674193 21656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:01.674686 21656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:01.675465 21656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:01.675505 21656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.675563 21656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:01.675592 21656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.683094 21656 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:42657
I20260923 14:40:01.683120 21783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:42657 every 8 connection(s)
I20260923 14:40:01.683825 21656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:40:01.691527 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 21656
I20260923 14:40:01.691654 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:40:01.691939 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.685356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:42591 (local address 127.2.218.62:39613)
0923 14:40:01.685480 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.685483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.686419 (+ 936us) server_negotiation.cc:407] Connection header received
0923 14:40:01.687677 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.687680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.687733 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.687822 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:01.688541 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.689022 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.690016 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.690292 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.690434 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.691359 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:01.691362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:01.691365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:01.691388 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:01.691405 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:01.691412 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:01.691496 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.691723 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.691784 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 14:40:01.691982 2920 external_mini_cluster.cc:1428] Setting key 9d39ee1153f1144f08fed88749b9e12e
I20260923 14:40:01.692654 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.685676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39613 (local address 127.2.218.1:42591)
0923 14:40:01.686300 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:01.686335 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:01.687464 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:01.687971 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:01.687978 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:01.688047 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:01.688422 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.688432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.689134 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.689137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.689903 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.689911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.690463 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.690467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.690611 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.691128 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:40:01.691130 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:01.691162 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:40:01.691216 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:01.691513 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:01.691528 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:01.691532 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 14:40:01.691655 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 14:40:01.691883 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
I20260923 14:40:01.693689 21784 heartbeater.cc:347] Connected to a master server at 127.2.218.62:39613
I20260923 14:40:01.693879 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39613
--builtin_ntp_servers=127.2.218.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:40:01.693907 21784 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:01.694351 21784 heartbeater.cc:511] Master 127.2.218.62:39613 requested a full tablet report, sending...
I20260923 14:40:01.695824 21596 ts_manager.cc:194] Registered new tserver with Master: 4738d16ad498426abbda6d21b3f3c6e5 (127.2.218.1:42657)
I20260923 14:40:01.697242 21596 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.1:42591
W20260923 14:40:01.789039 21787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:01.789232 21787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:01.789263 21787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:01.792608 21787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:01.792650 21787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:01.792696 21787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:40:01.796734 21787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:01.797426 21787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:01.798461 21787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:01.806221 21793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:01.806221 21795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:01.806238 21792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:01.806409 21787 server_base.cc:1034] running on GCE node
I20260923 14:40:01.806734 21787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:01.807284 21787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:01.808481 21787 hybrid_clock.cc:648] HybridClock initialized: now 1790174401808465 us; error 26 us; skew 500 ppm
I20260923 14:40:01.810065 21787 webserver.cc:460] Webserver started at http://127.2.218.2:33203/ using document root <none> and password file <none>
I20260923 14:40:01.810588 21787 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:01.810645 21787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:01.810858 21787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:01.812508 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "1fa61080e29b4c55adfcf4553ddc3e75"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "e2326baf4935b59077de641d6a6dd6b2"
server_key_iv: "7671809c7ade88e13e023d814c11c94b"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.812989 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "1fa61080e29b4c55adfcf4553ddc3e75"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "e2326baf4935b59077de641d6a6dd6b2"
server_key_iv: "7671809c7ade88e13e023d814c11c94b"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.815989 21787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 14:40:01.817888 21801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.818878 21787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:40:01.819015 21787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "1fa61080e29b4c55adfcf4553ddc3e75"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "e2326baf4935b59077de641d6a6dd6b2"
server_key_iv: "7671809c7ade88e13e023d814c11c94b"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.819123 21787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:01.831259 21787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:01.831753 21787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:01.831934 21787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:01.832423 21787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:01.833223 21787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:01.833304 21787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.833389 21787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:01.833441 21787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.841423 21787 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:35981
I20260923 14:40:01.841446 21914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:35981 every 8 connection(s)
I20260923 14:40:01.842137 21787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:40:01.849148 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.843595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:36571 (local address 127.2.218.62:39613)
0923 14:40:01.843746 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:01.843751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:01.844578 (+ 827us) server_negotiation.cc:407] Connection header received
0923 14:40:01.845618 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:01.845621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:01.845666 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:01.845736 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:01.846390 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.846836 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.847751 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.847950 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.848028 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.848796 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:01.848799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:01.848801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:01.848812 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:01.848831 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:01.848838 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:01.848895 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:01.849003 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 14:40:01.849050 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:40:01.849758 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:01.843912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39613 (local address 127.2.218.2:36571)
0923 14:40:01.844461 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:01.844492 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:01.845462 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:01.845854 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:01.845859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:01.845922 (+ 63us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:01.846290 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.846299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.846938 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.846940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.847652 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:01.847659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:01.848063 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:01.848065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:01.848171 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:01.848601 (+ 430us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:40:01.848603 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:01.848631 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:40:01.848677 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:01.848903 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:01.848908 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:01.848911 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:40:01.848984 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 14:40:01.849139 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":106,"threads_started":1}
I20260923 14:40:01.850018 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 21787
I20260923 14:40:01.850162 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:40:01.850425 2920 external_mini_cluster.cc:1428] Setting key c8184185631f9fba5df44e374047fc98
I20260923 14:40:01.850821 21915 heartbeater.cc:347] Connected to a master server at 127.2.218.62:39613
I20260923 14:40:01.851013 21915 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:01.851521 21915 heartbeater.cc:511] Master 127.2.218.62:39613 requested a full tablet report, sending...
I20260923 14:40:01.852272 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39613
--builtin_ntp_servers=127.2.218.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:40:01.852547 21596 ts_manager.cc:194] Registered new tserver with Master: 1fa61080e29b4c55adfcf4553ddc3e75 (127.2.218.2:35981)
I20260923 14:40:01.853106 21596 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.2:36571
W20260923 14:40:01.946960 21918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:01.947130 21918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:01.947161 21918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:01.950515 21918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:01.950555 21918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:01.950623 21918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:40:01.954537 21918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:01.955252 21918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:01.956278 21918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:01.964293 21923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:01.964304 21926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:01.964555 21924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:01.964712 21918 server_base.cc:1034] running on GCE node
I20260923 14:40:01.965173 21918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:01.965905 21918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:01.967105 21918 hybrid_clock.cc:648] HybridClock initialized: now 1790174401967066 us; error 51 us; skew 500 ppm
I20260923 14:40:01.969411 21918 webserver.cc:460] Webserver started at http://127.2.218.3:33585/ using document root <none> and password file <none>
I20260923 14:40:01.970017 21918 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:01.970115 21918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:01.970350 21918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:01.972761 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "b8e93b1c90924375b7768ff36db5a5cc"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "e98e6a199d70cbf828fd7579e8f0fdf2"
server_key_iv: "67d92f33d87dcfb00aa131a3ea624880"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.973472 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "b8e93b1c90924375b7768ff36db5a5cc"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "e98e6a199d70cbf828fd7579e8f0fdf2"
server_key_iv: "67d92f33d87dcfb00aa131a3ea624880"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.977852 21918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260923 14:40:01.980710 21932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:01.981959 21918 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:01.982075 21918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "b8e93b1c90924375b7768ff36db5a5cc"
format_stamp: "Formatted at 2026-09-23 14:40:01 on dist-test-slave-zl47"
server_key: "e98e6a199d70cbf828fd7579e8f0fdf2"
server_key_iv: "67d92f33d87dcfb00aa131a3ea624880"
server_key_version: "encryptionkey@0"
I20260923 14:40:01.982224 21918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:02.000427 21918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:02.001075 21918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:02.001286 21918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:02.001965 21918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:02.003127 21918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:02.003191 21918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:02.003247 21918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:02.003283 21918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:02.015182 21918 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:37307
I20260923 14:40:02.015182 22045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:37307 every 8 connection(s)
I20260923 14:40:02.016283 21918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:40:02.017531 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 21918
I20260923 14:40:02.017645 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:40:02.017982 2920 external_mini_cluster.cc:1428] Setting key c3a44033b75ae1d202d75f53c2dad7d8
I20260923 14:40:02.026589 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.018244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:60899 (local address 127.2.218.62:39613)
0923 14:40:02.018377 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:02.018382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:02.019203 (+ 821us) server_negotiation.cc:407] Connection header received
0923 14:40:02.021200 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:02.021203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:02.021245 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:02.021336 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:02.022472 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.023053 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.024722 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.024910 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.024979 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.026172 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:02.026174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:02.026176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:02.026186 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:02.026203 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:02.026209 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:02.026259 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:02.026456 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 14:40:02.026498 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 14:40:02.027638 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.018243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:39613 (local address 127.2.218.3:60899)
0923 14:40:02.019047 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:02.019101 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:02.020948 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:02.021503 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:02.021514 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:02.021632 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:02.022290 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.022311 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.023170 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.023176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.024602 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.024617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.025059 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.025065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.025277 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.025863 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:40:02.025866 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:02.025918 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:40:02.025992 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:02.026287 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:02.026297 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:02.026304 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 14:40:02.026455 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 14:40:02.026744 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":135,"threads_started":1}
I20260923 14:40:02.028663 22046 heartbeater.cc:347] Connected to a master server at 127.2.218.62:39613
I20260923 14:40:02.028896 22046 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:02.029482 22046 heartbeater.cc:511] Master 127.2.218.62:39613 requested a full tablet report, sending...
I20260923 14:40:02.030493 21596 ts_manager.cc:194] Registered new tserver with Master: b8e93b1c90924375b7768ff36db5a5cc (127.2.218.3:37307)
I20260923 14:40:02.031149 21596 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.3:60899
I20260923 14:40:02.031687 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:40:02.039296 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.033538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.2.218.62:39613)
0923 14:40:02.033642 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:02.033645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:02.034384 (+ 739us) server_negotiation.cc:407] Connection header received
0923 14:40:02.034515 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:02.034518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:02.034557 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:02.034649 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:02.035007 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.036016 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.037282 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.037631 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.037723 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.038694 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:02.038698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:02.038701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:02.038712 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:02.038730 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:02.038742 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:02.038892 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:02.039070 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 14:40:02.039129 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 14:40:02.042003 21595 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:40:02.044687 21595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:40:02.059496 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.053011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:35981 (local address 127.0.0.1:34974)
0923 14:40:02.054016 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:02.054032 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:02.054140 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:02.054793 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:02.054798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:02.054818 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:02.054951 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.054962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.056369 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.056372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.057945 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.057956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.059002 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.059006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.059121 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.059161 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 14:40:02.059243 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 14:40:02.059371 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":113,"threads_started":1}
I20260923 14:40:02.060503 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.053408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.2.218.2:35981)
0923 14:40:02.054382 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:02.054395 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:02.054419 (+ 24us) server_negotiation.cc:407] Connection header received
0923 14:40:02.054491 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:02.054497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:02.054643 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:02.054799 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:02.055080 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.056206 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.058090 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.058863 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.059998 (+ 1135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.060175 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:02.060223 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 14:40:02.060312 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":101,"threads_started":1}
I20260923 14:40:02.059731 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.052489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:37307 (local address 127.0.0.1:55436)
0923 14:40:02.053322 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:02.053375 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:02.053483 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:02.054986 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:02.054991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:02.055013 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:02.055098 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.055103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.057284 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.057288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.058401 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.058410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.059382 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.059386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.059466 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.059488 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 14:40:02.059540 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 14:40:02.059617 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":29,"thread_start_us":74,"threads_started":1}
I20260923 14:40:02.060761 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.053491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.2.218.1:42657)
0923 14:40:02.053786 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:02.053796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:02.054517 (+ 721us) server_negotiation.cc:407] Connection header received
0923 14:40:02.054602 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:02.054609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:02.054827 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:02.054970 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:02.055447 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.056933 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.058767 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.059679 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.059792 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.059999 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:02.060447 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 14:40:02.060557 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":119,"threads_started":1}
I20260923 14:40:02.061069 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.053084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:42657 (local address 127.0.0.1:32874)
0923 14:40:02.054331 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:02.054344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:02.054427 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:02.055129 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:02.055134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:02.055154 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:02.055273 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.055281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.057084 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.057090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.058616 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.058629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.060065 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.060070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.060187 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.060219 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 14:40:02.060290 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 14:40:02.060370 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":67,"thread_start_us":84,"threads_started":1}
I20260923 14:40:02.061262 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.052580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.2.218.3:37307)
0923 14:40:02.053783 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:02.053795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:02.053816 (+ 21us) server_negotiation.cc:407] Connection header received
0923 14:40:02.053908 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:02.053913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:02.054084 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:02.055233 (+ 1149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:02.055347 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.056391 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.058540 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.059256 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.060778 (+ 1522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.060955 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:02.061003 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 14:40:02.061107 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1118,"thread_start_us":120,"threads_started":1}
I20260923 14:40:02.063652 21849 tablet_service.cc:1467] Processing CreateTablet for tablet efa2ced6ad2b4f8bb616482bd9a77ea3 (DEFAULT_TABLE table=test-table [id=2070b927fa7c412493e620667218e677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:02.063668 21980 tablet_service.cc:1467] Processing CreateTablet for tablet efa2ced6ad2b4f8bb616482bd9a77ea3 (DEFAULT_TABLE table=test-table [id=2070b927fa7c412493e620667218e677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:02.063920 21718 tablet_service.cc:1467] Processing CreateTablet for tablet efa2ced6ad2b4f8bb616482bd9a77ea3 (DEFAULT_TABLE table=test-table [id=2070b927fa7c412493e620667218e677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:02.064584 21849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa2ced6ad2b4f8bb616482bd9a77ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:02.064584 21980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa2ced6ad2b4f8bb616482bd9a77ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:02.064900 21718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa2ced6ad2b4f8bb616482bd9a77ea3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:02.072801 22063 tablet_bootstrap.cc:492] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: Bootstrap starting.
I20260923 14:40:02.073593 22064 tablet_bootstrap.cc:492] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5: Bootstrap starting.
I20260923 14:40:02.074391 22065 tablet_bootstrap.cc:492] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: Bootstrap starting.
I20260923 14:40:02.075191 22063 tablet_bootstrap.cc:654] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:02.075541 22064 tablet_bootstrap.cc:654] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:02.076041 22063 log.cc:824] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:02.076200 22064 log.cc:824] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:02.076522 22065 tablet_bootstrap.cc:654] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:02.077224 22065 log.cc:824] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:02.077822 22063 tablet_bootstrap.cc:492] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: No bootstrap required, opened a new log
I20260923 14:40:02.077921 22064 tablet_bootstrap.cc:492] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5: No bootstrap required, opened a new log
I20260923 14:40:02.078016 22063 ts_tablet_manager.cc:1397] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:40:02.078094 22064 ts_tablet_manager.cc:1397] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:40:02.078879 22065 tablet_bootstrap.cc:492] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: No bootstrap required, opened a new log
I20260923 14:40:02.079030 22065 ts_tablet_manager.cc:1397] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 14:40:02.080710 22064 raft_consensus.cc:348] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e93b1c90924375b7768ff36db5a5cc" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37307 } } peers { permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 } } peers { permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 } }
I20260923 14:40:02.080904 22064 raft_consensus.cc:374] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:02.080950 22064 raft_consensus.cc:729] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4738d16ad498426abbda6d21b3f3c6e5, State: Initialized, Role: FOLLOWER
I20260923 14:40:02.081339 22064 consensus_queue.cc:260] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e93b1c90924375b7768ff36db5a5cc" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37307 } } peers { permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 } } peers { permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 } }
I20260923 14:40:02.081497 22063 raft_consensus.cc:348] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e93b1c90924375b7768ff36db5a5cc" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37307 } } peers { permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 } } peers { permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 } }
I20260923 14:40:02.081723 22063 raft_consensus.cc:374] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:02.081777 22063 raft_consensus.cc:729] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa61080e29b4c55adfcf4553ddc3e75, State: Initialized, Role: FOLLOWER
I20260923 14:40:02.082001 21784 heartbeater.cc:503] Master 127.2.218.62:39613 was elected leader, sending a full tablet report...
I20260923 14:40:02.082193 22064 ts_tablet_manager.cc:1428] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260923 14:40:02.082602 22063 consensus_queue.cc:260] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e93b1c90924375b7768ff36db5a5cc" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37307 } } peers { permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 } } peers { permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 } }
I20260923 14:40:02.082602 22065 raft_consensus.cc:348] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e93b1c90924375b7768ff36db5a5cc" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37307 } } peers { permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 } } peers { permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 } }
I20260923 14:40:02.082935 22065 raft_consensus.cc:374] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:02.082989 22065 raft_consensus.cc:729] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e93b1c90924375b7768ff36db5a5cc, State: Initialized, Role: FOLLOWER
I20260923 14:40:02.083346 22065 consensus_queue.cc:260] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e93b1c90924375b7768ff36db5a5cc" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37307 } } peers { permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 } } peers { permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 } }
I20260923 14:40:02.083546 22063 ts_tablet_manager.cc:1428] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:40:02.083813 21915 heartbeater.cc:503] Master 127.2.218.62:39613 was elected leader, sending a full tablet report...
I20260923 14:40:02.084048 22065 ts_tablet_manager.cc:1428] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:40:02.084308 22046 heartbeater.cc:503] Master 127.2.218.62:39613 was elected leader, sending a full tablet report...
W20260923 14:40:02.092936 21916 tablet.cc:2369] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:40:02.184846 21785 tablet.cc:2369] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:40:02.267238 22047 tablet.cc:2369] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:40:02.301411 22071 raft_consensus.cc:482] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:40:02.301577 22071 raft_consensus.cc:504] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e93b1c90924375b7768ff36db5a5cc" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37307 } } peers { permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 } } peers { permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 } }
I20260923 14:40:02.302606 22071 leader_election.cc:288] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fa61080e29b4c55adfcf4553ddc3e75 (127.2.218.2:35981), 4738d16ad498426abbda6d21b3f3c6e5 (127.2.218.1:42657)
I20260923 14:40:02.306635 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.302934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:54965 (local address 127.2.218.2:35981)
0923 14:40:02.303045 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:02.303049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:02.303473 (+ 424us) server_negotiation.cc:407] Connection header received
0923 14:40:02.303546 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:02.303550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:02.303599 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:02.303675 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:02.303883 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.304466 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.305619 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.306148 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.306235 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.306284 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:02.306456 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 14:40:02.306518 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:40:02.306634 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.302911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:49343 (local address 127.2.218.1:42657)
0923 14:40:02.303042 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:02.303048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:02.303066 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:40:02.303123 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:02.303129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:02.303188 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:02.303275 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:02.303566 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.304244 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.305497 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.306037 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.306205 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.306273 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:02.306456 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 14:40:02.306518 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:40:02.306633 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.302822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:42657 (local address 127.2.218.3:49343)
0923 14:40:02.302955 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:02.302972 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:02.303044 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:02.303297 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:02.303300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:02.303315 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:02.303442 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.303450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.304346 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.304349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.305381 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.305391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.306196 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.306212 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.306308 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.306344 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 14:40:02.306423 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 14:40:02.306504 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":19}
I20260923 14:40:02.306640 22072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.302822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:35981 (local address 127.2.218.3:54965)
0923 14:40:02.303369 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:02.303380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:02.303461 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:02.303687 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:02.303689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:02.303701 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:02.303785 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.303789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.304554 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.304557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.305514 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:02.305520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.306253 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.306255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:02.306346 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.306360 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 14:40:02.306423 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 14:40:02.306504 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":258,"thread_start_us":102,"threads_started":1}
I20260923 14:40:02.307255 21869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa2ced6ad2b4f8bb616482bd9a77ea3" candidate_uuid: "b8e93b1c90924375b7768ff36db5a5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" is_pre_election: true
I20260923 14:40:02.307541 21869 raft_consensus.cc:2454] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e93b1c90924375b7768ff36db5a5cc in term 0.
I20260923 14:40:02.307678 21738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa2ced6ad2b4f8bb616482bd9a77ea3" candidate_uuid: "b8e93b1c90924375b7768ff36db5a5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4738d16ad498426abbda6d21b3f3c6e5" is_pre_election: true
I20260923 14:40:02.307935 21738 raft_consensus.cc:2454] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8e93b1c90924375b7768ff36db5a5cc in term 0.
I20260923 14:40:02.307978 21934 leader_election.cc:302] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa61080e29b4c55adfcf4553ddc3e75, b8e93b1c90924375b7768ff36db5a5cc; no voters:
I20260923 14:40:02.308203 22071 raft_consensus.cc:2781] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:40:02.308285 22071 raft_consensus.cc:482] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:40:02.308331 22071 raft_consensus.cc:3037] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:02.309204 22071 raft_consensus.cc:504] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e93b1c90924375b7768ff36db5a5cc" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37307 } } peers { permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 } } peers { permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 } }
I20260923 14:40:02.309476 22071 leader_election.cc:288] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [CANDIDATE]: Term 1 election: Requested vote from peers 1fa61080e29b4c55adfcf4553ddc3e75 (127.2.218.2:35981), 4738d16ad498426abbda6d21b3f3c6e5 (127.2.218.1:42657)
I20260923 14:40:02.309751 21869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa2ced6ad2b4f8bb616482bd9a77ea3" candidate_uuid: "b8e93b1c90924375b7768ff36db5a5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa61080e29b4c55adfcf4553ddc3e75"
I20260923 14:40:02.309801 21738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa2ced6ad2b4f8bb616482bd9a77ea3" candidate_uuid: "b8e93b1c90924375b7768ff36db5a5cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4738d16ad498426abbda6d21b3f3c6e5"
I20260923 14:40:02.309849 21869 raft_consensus.cc:3037] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:02.309955 21738 raft_consensus.cc:3037] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:02.310779 21869 raft_consensus.cc:2454] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e93b1c90924375b7768ff36db5a5cc in term 1.
I20260923 14:40:02.311002 21738 raft_consensus.cc:2454] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8e93b1c90924375b7768ff36db5a5cc in term 1.
I20260923 14:40:02.311061 21934 leader_election.cc:302] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa61080e29b4c55adfcf4553ddc3e75, b8e93b1c90924375b7768ff36db5a5cc; no voters:
I20260923 14:40:02.311295 22071 raft_consensus.cc:2781] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:02.311491 22071 raft_consensus.cc:686] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 1 LEADER]: Becoming Leader. State: Replica: b8e93b1c90924375b7768ff36db5a5cc, State: Running, Role: LEADER
I20260923 14:40:02.311764 22071 consensus_queue.cc:237] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e93b1c90924375b7768ff36db5a5cc" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37307 } } peers { permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 } } peers { permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 } }
I20260923 14:40:02.314307 21595 catalog_manager.cc:5495] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc reported cstate change: term changed from 0 to 1, leader changed from <none> to b8e93b1c90924375b7768ff36db5a5cc (127.2.218.3). New cstate: current_term: 1 leader_uuid: "b8e93b1c90924375b7768ff36db5a5cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8e93b1c90924375b7768ff36db5a5cc" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 } health_report { overall_health: UNKNOWN } } }
I20260923 14:40:02.347157 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:02.343911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.2.218.3:37307)
0923 14:40:02.344034 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:02.344039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:02.344052 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:40:02.344112 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:02.344113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:02.344155 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:02.344257 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:02.344501 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.344934 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.345729 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:02.345944 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:02.346049 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:02.346399 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:02.346719 (+ 320us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:02.346893 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:02.346981 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 14:40:02.347010 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:40:02.351774 21869 raft_consensus.cc:1261] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75 [term 1 FOLLOWER]: Refusing update from remote peer b8e93b1c90924375b7768ff36db5a5cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:02.351773 21738 raft_consensus.cc:1261] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 4738d16ad498426abbda6d21b3f3c6e5 [term 1 FOLLOWER]: Refusing update from remote peer b8e93b1c90924375b7768ff36db5a5cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:02.352372 22071 consensus_queue.cc:1035] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa61080e29b4c55adfcf4553ddc3e75" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 35981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:02.352542 22073 consensus_queue.cc:1035] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4738d16ad498426abbda6d21b3f3c6e5" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:02.366328 21596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:40:02.366438 21596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:40:02.368598 21596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddf62dec5e4547cca1474d7b78438985: Sending DeleteTablet for 3 replicas of tablet efa2ced6ad2b4f8bb616482bd9a77ea3
I20260923 14:40:02.369212 21849 tablet_service.cc:1514] Processing DeleteTablet for tablet efa2ced6ad2b4f8bb616482bd9a77ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:02 UTC) from {username='slave'} at 127.0.0.1:34974
I20260923 14:40:02.369269 21718 tablet_service.cc:1514] Processing DeleteTablet for tablet efa2ced6ad2b4f8bb616482bd9a77ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:02 UTC) from {username='slave'} at 127.0.0.1:32874
I20260923 14:40:02.369328 21980 tablet_service.cc:1514] Processing DeleteTablet for tablet efa2ced6ad2b4f8bb616482bd9a77ea3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:02 UTC) from {username='slave'} at 127.0.0.1:55436
I20260923 14:40:02.369781 22085 tablet_replica.cc:330] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: stopping tablet replica
I20260923 14:40:02.370079 22085 raft_consensus.cc:2229] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:40:02.370234 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 21656
I20260923 14:40:02.370359 22085 raft_consensus.cc:2258] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:40:02.371919 22085 ts_tablet_manager.cc:1905] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:40:02.374784 22087 tablet_replica.cc:330] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: stopping tablet replica
I20260923 14:40:02.374969 22085 ts_tablet_manager.cc:1918] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:40:02.375056 22085 log.cc:1196] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/efa2ced6ad2b4f8bb616482bd9a77ea3
I20260923 14:40:02.375216 22087 raft_consensus.cc:2229] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:40:02.375325 22085 ts_tablet_manager.cc:1939] T efa2ced6ad2b4f8bb616482bd9a77ea3 P 1fa61080e29b4c55adfcf4553ddc3e75: Deleting consensus metadata
I20260923 14:40:02.375578 22087 raft_consensus.cc:2258] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:40:02.376425 21581 catalog_manager.cc:4841] TS 1fa61080e29b4c55adfcf4553ddc3e75 (127.2.218.2:35981): tablet efa2ced6ad2b4f8bb616482bd9a77ea3 (table test-table [id=2070b927fa7c412493e620667218e677]) successfully deleted
I20260923 14:40:02.376818 22087 ts_tablet_manager.cc:1905] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 14:40:02.377782 21580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:42657 (error 108)
W20260923 14:40:02.378926 22057 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:02.378667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:42657 (local address 127.0.0.1:32880)
0923 14:40:02.378781 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:02.378841 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:42657: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 14:40:02.379030 21580 catalog_manager.cc:4584] TS 4738d16ad498426abbda6d21b3f3c6e5 (127.2.218.1:42657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet efa2ced6ad2b4f8bb616482bd9a77ea3: Network error: Client connection negotiation failed: client connection to 127.2.218.1:42657: connect: Connection reset by peer (error 104)
I20260923 14:40:02.379159 22087 ts_tablet_manager.cc:1918] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:40:02.379242 22087 log.cc:1196] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/efa2ced6ad2b4f8bb616482bd9a77ea3
I20260923 14:40:02.379321 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 21787
I20260923 14:40:02.381043 22087 ts_tablet_manager.cc:1939] T efa2ced6ad2b4f8bb616482bd9a77ea3 P b8e93b1c90924375b7768ff36db5a5cc: Deleting consensus metadata
I20260923 14:40:02.381937 21580 catalog_manager.cc:4841] TS b8e93b1c90924375b7768ff36db5a5cc (127.2.218.3:37307): tablet efa2ced6ad2b4f8bb616482bd9a77ea3 (table test-table [id=2070b927fa7c412493e620667218e677]) successfully deleted
I20260923 14:40:02.387377 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 21918
I20260923 14:40:02.391789 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 21565
2026-09-23T14:40:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1114 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:02 dist-test-slave-zl47 krb5kdc[22091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:02 dist-test-slave-zl47 krb5kdc[22091](info): set up 2 sockets
Sep 23 14:40:02 dist-test-slave-zl47 krb5kdc[22091](info): commencing operation
krb5kdc: starting...
W20260923 14:40:03.282358 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.871s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:03 dist-test-slave-zl47 krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:40:03.382457 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:37825
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:37825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:03.473678 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:03.473905 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:03.473986 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:03.477406 22104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:03.477483 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:03.477523 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:03.477572 22104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:03.477603 22104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:03.482085 22104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:37825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:37825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:03.482909 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:03.483759 22104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:03.491410 22109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:03.491571 22110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:03.491571 22112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:03.491762 22104 server_base.cc:1034] running on GCE node
I20260923 14:40:03.492266 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:03.493196 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:03.494376 22104 hybrid_clock.cc:648] HybridClock initialized: now 1790174403494365 us; error 26 us; skew 500 ppm
Sep 23 14:40:03 dist-test-slave-zl47 krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:03.497138 22104 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:40:03.497984 22104 webserver.cc:460] Webserver started at http://127.2.218.62:34761/ using document root <none> and password file <none>
I20260923 14:40:03.498482 22104 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:03.498566 22104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:03.498767 22104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:03.500504 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "0662792aa29949d59b9ae8bd18c417c0"
format_stamp: "Formatted at 2026-09-23 14:40:03 on dist-test-slave-zl47"
server_key: "8975fc3d75f0a03a87155698caf91504"
server_key_iv: "118f195019be2251b02f43fd01f019dc"
server_key_version: "encryptionkey@0"
I20260923 14:40:03.500982 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "0662792aa29949d59b9ae8bd18c417c0"
format_stamp: "Formatted at 2026-09-23 14:40:03 on dist-test-slave-zl47"
server_key: "8975fc3d75f0a03a87155698caf91504"
server_key_iv: "118f195019be2251b02f43fd01f019dc"
server_key_version: "encryptionkey@0"
I20260923 14:40:03.504081 22104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 14:40:03.505934 22119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:03.507176 22104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:03.507301 22104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "0662792aa29949d59b9ae8bd18c417c0"
format_stamp: "Formatted at 2026-09-23 14:40:03 on dist-test-slave-zl47"
server_key: "8975fc3d75f0a03a87155698caf91504"
server_key_iv: "118f195019be2251b02f43fd01f019dc"
server_key_version: "encryptionkey@0"
I20260923 14:40:03.507375 22104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:03.528096 22104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:03.530253 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:03.530370 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:03.535982 22104 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:37825
I20260923 14:40:03.535991 22171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:37825 every 8 connection(s)
I20260923 14:40:03.536726 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:03.537106 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 22104
I20260923 14:40:03.537228 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:03.537448 2920 external_mini_cluster.cc:1428] Setting key a35fd6175fda8a10ad3f7cb2e0d33f2e
I20260923 14:40:03.539191 22172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:03.544672 22172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0: Bootstrap starting.
Sep 23 14:40:03 dist-test-slave-zl47 krb5kdc[22091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:03.546689 22172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:03.547292 22172 log.cc:824] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:03.549075 22172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0: No bootstrap required, opened a new log
I20260923 14:40:03.549613 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:03.538585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.2.218.62:37825)
0923 14:40:03.538919 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:03.538939 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:03.538966 (+ 27us) server_negotiation.cc:407] Connection header received
0923 14:40:03.539541 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:03.539555 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:03.539810 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:03.540033 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:03.540836 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:03.541731 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:03.542812 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:03.543186 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:03.543284 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:03.545617 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:03.545621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:03.545631 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:03.545652 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:03.547100 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:03.547517 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:03.547521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:03.547525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:03.547615 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:03.547908 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:03.547911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:03.547912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:03.548287 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:03.548463 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:03.548654 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 14:40:03.548884 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":118,"threads_started":1}
I20260923 14:40:03.552284 22172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0662792aa29949d59b9ae8bd18c417c0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37825 } }
I20260923 14:40:03.552469 22172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:03.552521 22172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0662792aa29949d59b9ae8bd18c417c0, State: Initialized, Role: FOLLOWER
I20260923 14:40:03.552842 22172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0662792aa29949d59b9ae8bd18c417c0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37825 } }
I20260923 14:40:03.552973 22172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:03.553009 22172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:03.553082 22172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:03.553875 22172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0662792aa29949d59b9ae8bd18c417c0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37825 } }
I20260923 14:40:03.554240 22172 leader_election.cc:302] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0662792aa29949d59b9ae8bd18c417c0; no voters:
I20260923 14:40:03.554522 22172 leader_election.cc:288] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:03.554661 22177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:03.554898 22177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [term 1 LEADER]: Becoming Leader. State: Replica: 0662792aa29949d59b9ae8bd18c417c0, State: Running, Role: LEADER
I20260923 14:40:03.555156 22177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0662792aa29949d59b9ae8bd18c417c0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37825 } }
I20260923 14:40:03.555526 22172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:03.556671 22179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0662792aa29949d59b9ae8bd18c417c0. Latest consensus state: current_term: 1 leader_uuid: "0662792aa29949d59b9ae8bd18c417c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0662792aa29949d59b9ae8bd18c417c0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37825 } } }
I20260923 14:40:03.556797 22179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:03.557657 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0662792aa29949d59b9ae8bd18c417c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0662792aa29949d59b9ae8bd18c417c0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 37825 } } }
I20260923 14:40:03.557762 22186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:03.557794 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:03.559921 22186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:03.564083 22186 catalog_manager.cc:1347] Generated new cluster ID: e19a82a574454c23a5515b9efa61840f
I20260923 14:40:03.564136 22186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:03.571875 22186 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:03.572592 22186 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:03.582106 22186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0: Generated new TSK 0
I20260923 14:40:03.582602 22186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.218.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
Entry for principal kudu/127.2.218.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab.
I20260923 14:40:03.613229 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37825
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:03.703929 22198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:03.704147 22198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:03.704211 22198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:03.707896 22198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:03.707963 22198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:03.708034 22198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:40:03.712368 22198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.1.keytab
--principal=kudu/127.2.218.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:03.713116 22198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:03.713893 22198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:03.721942 22204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:03.722002 22203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:03.721980 22198 server_base.cc:1034] running on GCE node
W20260923 14:40:03.721956 22206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:03.722594 22198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:03.723151 22198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:03.724329 22198 hybrid_clock.cc:648] HybridClock initialized: now 1790174403724317 us; error 26 us; skew 500 ppm
Sep 23 14:40:03 dist-test-slave-zl47 krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:03.726780 22198 init.cc:377] Logged in from keytab as kudu/127.2.218.1@KRBTEST.COM (short username kudu)
I20260923 14:40:03.727460 22198 webserver.cc:460] Webserver started at http://127.2.218.1:33789/ using document root <none> and password file <none>
I20260923 14:40:03.727921 22198 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:03.728003 22198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:03.728195 22198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:03.729800 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "e2d9ed4425a54e56a6812323f06d43ed"
format_stamp: "Formatted at 2026-09-23 14:40:03 on dist-test-slave-zl47"
server_key: "a203457d34758d1326a89d30e6b7825f"
server_key_iv: "5069b4aa20002ee11a6e82801e78ddc8"
server_key_version: "encryptionkey@0"
I20260923 14:40:03.730412 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "e2d9ed4425a54e56a6812323f06d43ed"
format_stamp: "Formatted at 2026-09-23 14:40:03 on dist-test-slave-zl47"
server_key: "a203457d34758d1326a89d30e6b7825f"
server_key_iv: "5069b4aa20002ee11a6e82801e78ddc8"
server_key_version: "encryptionkey@0"
I20260923 14:40:03.733667 22198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:40:03.735850 22213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:03.736909 22198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:03.737052 22198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "e2d9ed4425a54e56a6812323f06d43ed"
format_stamp: "Formatted at 2026-09-23 14:40:03 on dist-test-slave-zl47"
server_key: "a203457d34758d1326a89d30e6b7825f"
server_key_iv: "5069b4aa20002ee11a6e82801e78ddc8"
server_key_version: "encryptionkey@0"
I20260923 14:40:03.737159 22198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:03.749706 22198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:03.752040 22198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:03.752229 22198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:03.752692 22198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:03.753368 22198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:03.753437 22198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:03.753499 22198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:03.753556 22198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:03.761713 22198 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:36885
I20260923 14:40:03.761732 22326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:36885 every 8 connection(s)
I20260923 14:40:03.762450 22198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:40:03.767733 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 22198
I20260923 14:40:03.767843 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:40:03.768100 2920 external_mini_cluster.cc:1428] Setting key 88296f571e5fa7390c82b71acc9da875
Sep 23 14:40:03 dist-test-slave-zl47 krb5kdc[22091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.1@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:03.773799 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:03.763938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:48301 (local address 127.2.218.62:37825)
0923 14:40:03.764061 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:03.764065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:03.764670 (+ 605us) server_negotiation.cc:407] Connection header received
0923 14:40:03.765381 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:03.765384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:03.765437 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:03.765533 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:03.766548 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:03.767033 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:03.768103 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:03.768353 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:03.768459 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:03.770636 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:03.770642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:03.770647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:03.770676 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:03.771868 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:03.772388 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:03.772390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:03.772391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:03.772464 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:03.772793 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:03.772795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:03.772796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:03.772929 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:03.772993 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:03.773591 (+ 598us) server_negotiation.cc:299] Negotiation successful
0923 14:40:03.773690 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:40:03.774899 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:03.764157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:37825 (local address 127.2.218.1:48301)
0923 14:40:03.764547 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:03.764580 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:03.765207 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:03.765675 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:03.765681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:03.765961 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:03.766422 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:03.766434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:03.767152 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:03.767171 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:03.767993 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:03.768002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:03.768501 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:03.768504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:03.768674 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:03.769115 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:03.769127 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:03.770498 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:03.771969 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:03.771976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:03.771989 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:03.772301 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:03.772545 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:03.772550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:03.772554 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:03.772713 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:03.773025 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:03.773033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:03.773319 (+ 286us) client_negotiation.cc:769] Sending connection context
0923 14:40:03.773550 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 14:40:03.773864 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
I20260923 14:40:03.776101 22327 heartbeater.cc:347] Connected to a master server at 127.2.218.62:37825
I20260923 14:40:03.776283 22327 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:03.776654 22327 heartbeater.cc:511] Master 127.2.218.62:37825 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.218.2@KRBTEST.COM; defaulting to no policy
I20260923 14:40:03.777889 22136 ts_manager.cc:194] Registered new tserver with Master: e2d9ed4425a54e56a6812323f06d43ed (127.2.218.1:36885)
I20260923 14:40:03.778934 22136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.1@KRBTEST.COM'} at 127.2.218.1:48301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
Entry for principal kudu/127.2.218.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab.
I20260923 14:40:03.790942 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37825
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:03.879334 22332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:03.879539 22332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:03.879613 22332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:03.882903 22332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:03.882970 22332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:03.883047 22332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:40:03.887280 22332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.2.keytab
--principal=kudu/127.2.218.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:03.888051 22332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:03.888827 22332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:03.896353 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:03.896380 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:03.896440 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:03.896728 22332 server_base.cc:1034] running on GCE node
I20260923 14:40:03.897059 22332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:03.897578 22332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:03.898747 22332 hybrid_clock.cc:648] HybridClock initialized: now 1790174403898736 us; error 31 us; skew 500 ppm
Sep 23 14:40:03 dist-test-slave-zl47 krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:03.901008 22332 init.cc:377] Logged in from keytab as kudu/127.2.218.2@KRBTEST.COM (short username kudu)
I20260923 14:40:03.901700 22332 webserver.cc:460] Webserver started at http://127.2.218.2:39589/ using document root <none> and password file <none>
I20260923 14:40:03.902215 22332 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:03.902297 22332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:03.902496 22332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:03.904096 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "027938c212a04dcdbcfca83bf772da16"
format_stamp: "Formatted at 2026-09-23 14:40:03 on dist-test-slave-zl47"
server_key: "e89522c2ac244de84bf2b567fae0a7ec"
server_key_iv: "5d543fbb3c33927770ba3b9bc23845fa"
server_key_version: "encryptionkey@0"
I20260923 14:40:03.904634 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "027938c212a04dcdbcfca83bf772da16"
format_stamp: "Formatted at 2026-09-23 14:40:03 on dist-test-slave-zl47"
server_key: "e89522c2ac244de84bf2b567fae0a7ec"
server_key_iv: "5d543fbb3c33927770ba3b9bc23845fa"
server_key_version: "encryptionkey@0"
I20260923 14:40:03.907619 22332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:40:03.909838 22347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:03.910974 22332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:03.911079 22332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "027938c212a04dcdbcfca83bf772da16"
format_stamp: "Formatted at 2026-09-23 14:40:03 on dist-test-slave-zl47"
server_key: "e89522c2ac244de84bf2b567fae0a7ec"
server_key_iv: "5d543fbb3c33927770ba3b9bc23845fa"
server_key_version: "encryptionkey@0"
I20260923 14:40:03.911183 22332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:03.930889 22332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:03.933377 22332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:03.933535 22332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:03.933969 22332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:03.934628 22332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:03.934688 22332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:03.934742 22332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:03.934778 22332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:03.942445 22332 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:40939
I20260923 14:40:03.942500 22460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:40939 every 8 connection(s)
I20260923 14:40:03.943167 22332 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:40:03.945175 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 22332
I20260923 14:40:03.945297 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:40:03.945580 2920 external_mini_cluster.cc:1428] Setting key c2bf08e8860e67c261d89f4dd0ca8dc6
Sep 23 14:40:03 dist-test-slave-zl47 krb5kdc[22091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174403, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.2@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.3@KRBTEST.COM; defaulting to no policy
I20260923 14:40:03.956005 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:03.944747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:41005 (local address 127.2.218.62:37825)
0923 14:40:03.944862 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:03.944865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:03.945674 (+ 809us) server_negotiation.cc:407] Connection header received
0923 14:40:03.946683 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:03.946687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:03.946735 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:03.946859 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:03.948125 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:03.948610 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:03.949628 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:03.949858 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:03.949970 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:03.952309 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:03.952313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:03.952316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:03.952340 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:03.953691 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:03.954285 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:03.954289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:03.954291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:03.954566 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:03.954916 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:03.954919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:03.954921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:03.955151 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:03.955275 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:03.955748 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 14:40:03.955853 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:40:03.956692 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:03.945035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:37825 (local address 127.2.218.2:41005)
0923 14:40:03.945495 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:03.945559 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:03.946484 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:03.947028 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:03.947038 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:03.947445 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:03.947981 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:03.947999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:03.948735 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:03.948739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:03.949501 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:03.949510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:03.949999 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:03.950003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:03.950209 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:03.950710 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:03.950734 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:03.952153 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:03.953820 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:03.953826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:03.953838 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:03.954140 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:03.954682 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:03.954685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:03.954687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:03.954814 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:03.955260 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:03.955265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:03.955523 (+ 258us) client_negotiation.cc:769] Sending connection context
0923 14:40:03.955694 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 14:40:03.955955 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.3@KRBTEST.COM" created.
I20260923 14:40:03.957656 22461 heartbeater.cc:347] Connected to a master server at 127.2.218.62:37825
I20260923 14:40:03.957851 22461 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:03.958280 22461 heartbeater.cc:511] Master 127.2.218.62:37825 requested a full tablet report, sending...
I20260923 14:40:03.959148 22136 ts_manager.cc:194] Registered new tserver with Master: 027938c212a04dcdbcfca83bf772da16 (127.2.218.2:40939)
I20260923 14:40:03.959645 22136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.2@KRBTEST.COM'} at 127.2.218.2:41005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
Entry for principal kudu/127.2.218.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab.
I20260923 14:40:03.969147 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37825
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:32895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:04.073287 22466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:04.073503 22466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:04.073578 22466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:04.077064 22466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:04.077107 22466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:04.077191 22466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:40:04.081758 22466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:32895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.3.keytab
--principal=kudu/127.2.218.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:37825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:04.082571 22466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:04.083359 22466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:04.091290 22474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:04.091282 22471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:04.091279 22472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:04.091380 22466 server_base.cc:1034] running on GCE node
I20260923 14:40:04.091838 22466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:04.092339 22466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:04.093523 22466 hybrid_clock.cc:648] HybridClock initialized: now 1790174404093488 us; error 26 us; skew 500 ppm
Sep 23 14:40:04 dist-test-slave-zl47 krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:04.096050 22466 init.cc:377] Logged in from keytab as kudu/127.2.218.3@KRBTEST.COM (short username kudu)
I20260923 14:40:04.096756 22466 webserver.cc:460] Webserver started at http://127.2.218.3:46665/ using document root <none> and password file <none>
I20260923 14:40:04.097208 22466 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:04.097285 22466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:04.097472 22466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:04.099349 22466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "c53e2d59dd814ce9af21a2ea2ee966da"
format_stamp: "Formatted at 2026-09-23 14:40:04 on dist-test-slave-zl47"
server_key: "605fee225b25dcabac01eefc82e5edd4"
server_key_iv: "62207d0b17f7ba1b13827f634ccdf021"
server_key_version: "encryptionkey@0"
I20260923 14:40:04.099833 22466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "c53e2d59dd814ce9af21a2ea2ee966da"
format_stamp: "Formatted at 2026-09-23 14:40:04 on dist-test-slave-zl47"
server_key: "605fee225b25dcabac01eefc82e5edd4"
server_key_iv: "62207d0b17f7ba1b13827f634ccdf021"
server_key_version: "encryptionkey@0"
I20260923 14:40:04.102864 22466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:40:04.104820 22481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:04.105821 22466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:04.105974 22466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "c53e2d59dd814ce9af21a2ea2ee966da"
format_stamp: "Formatted at 2026-09-23 14:40:04 on dist-test-slave-zl47"
server_key: "605fee225b25dcabac01eefc82e5edd4"
server_key_iv: "62207d0b17f7ba1b13827f634ccdf021"
server_key_version: "encryptionkey@0"
I20260923 14:40:04.106148 22466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:04.121727 22466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:04.124396 22466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:04.124555 22466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:04.124996 22466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:04.125638 22466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:04.125699 22466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:04.125756 22466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:04.125792 22466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:04.134596 22466 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:33049
I20260923 14:40:04.134631 22594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:33049 every 8 connection(s)
I20260923 14:40:04.135269 22466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:40:04.135545 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 22466
I20260923 14:40:04.135677 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:40:04.135998 2920 external_mini_cluster.cc:1428] Setting key 4a75c408710ff681862bc4d6a8cfc7fe
Sep 23 14:40:04 dist-test-slave-zl47 krb5kdc[22091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174404, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.3@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:04.146776 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.136762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:36835 (local address 127.2.218.62:37825)
0923 14:40:04.136858 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:04.136868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:04.137750 (+ 882us) server_negotiation.cc:407] Connection header received
0923 14:40:04.138667 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:04.138669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:04.138722 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:04.138780 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:04.139729 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.140265 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.141320 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.141509 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.141580 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.143592 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:04.143594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:04.143596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:04.143615 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:04.144841 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:04.145247 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:04.145249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:04.145250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:04.145311 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:04.145582 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:04.145584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:04.145585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:04.145749 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:04.145843 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:04.146496 (+ 653us) server_negotiation.cc:299] Negotiation successful
0923 14:40:04.146612 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 14:40:04.147529 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.137120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:37825 (local address 127.2.218.3:36835)
0923 14:40:04.137600 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:04.137646 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:04.138479 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:04.138920 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:04.138925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:04.139242 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:04.139612 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.139623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.140370 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.140373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.141212 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.141222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.141653 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.141657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.141800 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.142253 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:04.142267 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:04.143457 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:04.144940 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:04.144944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:04.144952 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:04.145168 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:04.145387 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:04.145389 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:04.145390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:04.145508 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:04.145857 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:04.145864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:04.146176 (+ 312us) client_negotiation.cc:769] Sending connection context
0923 14:40:04.146452 (+ 276us) client_negotiation.cc:240] Negotiation successful
0923 14:40:04.146778 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":118,"threads_started":1}
I20260923 14:40:04.148494 22595 heartbeater.cc:347] Connected to a master server at 127.2.218.62:37825
I20260923 14:40:04.148693 22595 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:04.149164 22595 heartbeater.cc:511] Master 127.2.218.62:37825 requested a full tablet report, sending...
I20260923 14:40:04.150128 22136 ts_manager.cc:194] Registered new tserver with Master: c53e2d59dd814ce9af21a2ea2ee966da (127.2.218.3:33049)
I20260923 14:40:04.150612 22136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.218.3@KRBTEST.COM'} at 127.2.218.3:36835
I20260923 14:40:04.161018 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:40:04.168853 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.162698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.2.218.62:37825)
0923 14:40:04.162797 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:04.162800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:04.162990 (+ 190us) server_negotiation.cc:407] Connection header received
0923 14:40:04.163123 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:04.163126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:04.163159 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:04.163237 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:04.163804 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.164217 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.165006 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.165209 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.165303 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.166437 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:04.166441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:04.166443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:04.166463 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:04.167463 (+ 1000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:04.167813 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:04.167816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:04.167818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:04.167898 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:04.168136 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:04.168139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:04.168140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:04.168314 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:04.168442 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:04.168636 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 14:40:04.168728 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:40:04.171087 22136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:40:04.173552 22136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:40:04.186151 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.180772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:33049 (local address 127.0.0.1:57764)
0923 14:40:04.181335 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:04.181349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:04.181439 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:04.182068 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:04.182070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:04.182087 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:04.182197 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.182202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.183649 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.183653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.184626 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.184638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.185635 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.185639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.185735 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.185769 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 14:40:04.185896 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 14:40:04.185999 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":78,"thread_start_us":80,"threads_started":1}
I20260923 14:40:04.186156 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.181306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.2.218.3:33049)
0923 14:40:04.181610 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:04.181619 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:04.181633 (+ 14us) server_negotiation.cc:407] Connection header received
0923 14:40:04.181681 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:04.181685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:04.181815 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:04.182331 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:04.182445 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.183445 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.184778 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.185513 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.185632 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.185754 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:04.185922 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 14:40:04.185998 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":95,"threads_started":1}
I20260923 14:40:04.186383 22609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.181259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:36885 (local address 127.0.0.1:59622)
0923 14:40:04.181617 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:04.181628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:04.181697 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:04.182537 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:04.182541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:04.182561 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:04.182637 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.182642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.183809 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.183812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.185049 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.185059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.186025 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.186028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.186135 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.186154 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 14:40:04.186200 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 14:40:04.186268 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":74,"thread_start_us":73,"threads_started":1}
I20260923 14:40:04.186729 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.180628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40939 (local address 127.0.0.1:53422)
0923 14:40:04.180913 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:04.180954 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:04.181059 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:04.181893 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:04.181900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:04.181921 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:04.182053 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.182060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.183636 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.183639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.185049 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.185059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.186025 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.186028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.186329 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.186345 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 14:40:04.186424 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 14:40:04.186490 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":331,"thread_start_us":69,"threads_started":1}
I20260923 14:40:04.186944 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.181627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.2.218.1:36885)
0923 14:40:04.182057 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:04.182068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:04.182087 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:40:04.182173 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:04.182178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:04.182377 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:04.182757 (+ 380us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:04.182872 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.183650 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.185182 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.185902 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.186523 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.186650 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:04.186695 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 14:40:04.186781 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":93,"threads_started":1}
I20260923 14:40:04.187191 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.180995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.2.218.2:40939)
0923 14:40:04.181451 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:04.181462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:04.181483 (+ 21us) server_negotiation.cc:407] Connection header received
0923 14:40:04.181557 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:04.181562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:04.181729 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:04.182219 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:04.182408 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.183445 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.185181 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.185896 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.186065 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.186221 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:04.186950 (+ 729us) server_negotiation.cc:299] Negotiation successful
0923 14:40:04.187036 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":146,"threads_started":1}
I20260923 14:40:04.188936 22529 tablet_service.cc:1467] Processing CreateTablet for tablet b27d9369022e4c81b0d521edb53bd1c4 (DEFAULT_TABLE table=test-table [id=66da8bd03dba4651bbc238411b353d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:04.189703 22529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27d9369022e4c81b0d521edb53bd1c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:04.189411 22395 tablet_service.cc:1467] Processing CreateTablet for tablet b27d9369022e4c81b0d521edb53bd1c4 (DEFAULT_TABLE table=test-table [id=66da8bd03dba4651bbc238411b353d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:04.189409 22261 tablet_service.cc:1467] Processing CreateTablet for tablet b27d9369022e4c81b0d521edb53bd1c4 (DEFAULT_TABLE table=test-table [id=66da8bd03dba4651bbc238411b353d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:04.190413 22261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27d9369022e4c81b0d521edb53bd1c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:04.190413 22395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27d9369022e4c81b0d521edb53bd1c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:04.195145 22612 tablet_bootstrap.cc:492] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: Bootstrap starting.
I20260923 14:40:04.197124 22612 tablet_bootstrap.cc:654] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:04.197131 22614 tablet_bootstrap.cc:492] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: Bootstrap starting.
I20260923 14:40:04.198180 22612 log.cc:824] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:04.198443 22613 tablet_bootstrap.cc:492] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed: Bootstrap starting.
I20260923 14:40:04.199304 22614 tablet_bootstrap.cc:654] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:04.199873 22612 tablet_bootstrap.cc:492] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: No bootstrap required, opened a new log
I20260923 14:40:04.200073 22612 ts_tablet_manager.cc:1397] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:40:04.200107 22614 log.cc:824] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:04.200704 22613 tablet_bootstrap.cc:654] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:04.201407 22613 log.cc:824] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:04.201887 22614 tablet_bootstrap.cc:492] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: No bootstrap required, opened a new log
I20260923 14:40:04.202040 22614 ts_tablet_manager.cc:1397] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:40:04.202992 22613 tablet_bootstrap.cc:492] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed: No bootstrap required, opened a new log
I20260923 14:40:04.202911 22612 raft_consensus.cc:348] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 } } peers { permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 } } peers { permanent_uuid: "e2d9ed4425a54e56a6812323f06d43ed" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36885 } }
I20260923 14:40:04.203128 22612 raft_consensus.cc:374] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:04.203135 22613 ts_tablet_manager.cc:1397] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:40:04.203186 22612 raft_consensus.cc:729] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c53e2d59dd814ce9af21a2ea2ee966da, State: Initialized, Role: FOLLOWER
I20260923 14:40:04.203565 22612 consensus_queue.cc:260] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 } } peers { permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 } } peers { permanent_uuid: "e2d9ed4425a54e56a6812323f06d43ed" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36885 } }
I20260923 14:40:04.204083 22595 heartbeater.cc:503] Master 127.2.218.62:37825 was elected leader, sending a full tablet report...
I20260923 14:40:04.204387 22612 ts_tablet_manager.cc:1428] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260923 14:40:04.205181 22614 raft_consensus.cc:348] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 } } peers { permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 } } peers { permanent_uuid: "e2d9ed4425a54e56a6812323f06d43ed" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36885 } }
I20260923 14:40:04.205411 22614 raft_consensus.cc:374] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:04.205458 22614 raft_consensus.cc:729] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027938c212a04dcdbcfca83bf772da16, State: Initialized, Role: FOLLOWER
I20260923 14:40:04.205796 22614 consensus_queue.cc:260] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 } } peers { permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 } } peers { permanent_uuid: "e2d9ed4425a54e56a6812323f06d43ed" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36885 } }
I20260923 14:40:04.206434 22461 heartbeater.cc:503] Master 127.2.218.62:37825 was elected leader, sending a full tablet report...
I20260923 14:40:04.206357 22613 raft_consensus.cc:348] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 } } peers { permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 } } peers { permanent_uuid: "e2d9ed4425a54e56a6812323f06d43ed" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36885 } }
I20260923 14:40:04.206593 22613 raft_consensus.cc:374] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:04.206641 22613 raft_consensus.cc:729] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d9ed4425a54e56a6812323f06d43ed, State: Initialized, Role: FOLLOWER
I20260923 14:40:04.206700 22614 ts_tablet_manager.cc:1428] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260923 14:40:04.207029 22613 consensus_queue.cc:260] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 } } peers { permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 } } peers { permanent_uuid: "e2d9ed4425a54e56a6812323f06d43ed" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36885 } }
I20260923 14:40:04.207629 22327 heartbeater.cc:503] Master 127.2.218.62:37825 was elected leader, sending a full tablet report...
I20260923 14:40:04.207880 22613 ts_tablet_manager.cc:1428] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260923 14:40:04.263360 22328 tablet.cc:2369] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:40:04.386336 22596 tablet.cc:2369] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:40:04.426259 22620 raft_consensus.cc:482] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:40:04.426431 22620 raft_consensus.cc:504] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 } } peers { permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 } } peers { permanent_uuid: "e2d9ed4425a54e56a6812323f06d43ed" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36885 } }
I20260923 14:40:04.427479 22620 leader_election.cc:288] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 027938c212a04dcdbcfca83bf772da16 (127.2.218.2:40939), c53e2d59dd814ce9af21a2ea2ee966da (127.2.218.3:33049)
I20260923 14:40:04.430928 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.427621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40939 (local address 127.2.218.1:53705)
0923 14:40:04.427849 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:04.427873 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:04.427970 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:04.428239 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:04.428241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:04.428261 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:04.428387 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.428395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.429106 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.429110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.429845 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.429853 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.430499 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.430503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.430612 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.430639 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 14:40:04.430702 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 14:40:04.430751 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260923 14:40:04.430948 22608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.427697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:53705 (local address 127.2.218.2:40939)
0923 14:40:04.427873 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:04.427878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:04.427948 (+ 70us) server_negotiation.cc:407] Connection header received
0923 14:40:04.428075 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:04.428079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:04.428142 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:04.428221 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:04.428488 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.429007 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.429970 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.430390 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.430471 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.430543 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:04.430740 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 14:40:04.430818 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 14:40:04.431001 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.427697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:33049 (local address 127.2.218.1:35705)
0923 14:40:04.428075 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:04.428088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:04.428194 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:04.428418 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:04.428420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:04.428451 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:04.428570 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.428576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.429209 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.429211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.429896 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.429928 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.430578 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.430581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.430682 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.430728 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 14:40:04.430804 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 14:40:04.430864 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
I20260923 14:40:04.431005 22610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.427877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:35705 (local address 127.2.218.3:33049)
0923 14:40:04.427981 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:04.427985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:04.428172 (+ 187us) server_negotiation.cc:407] Connection header received
0923 14:40:04.428275 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:04.428277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:04.428319 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:04.428399 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:04.428672 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.429106 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.430034 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.430459 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.430551 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.430608 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:04.430837 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 14:40:04.430896 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 14:40:04.431489 22415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27d9369022e4c81b0d521edb53bd1c4" candidate_uuid: "e2d9ed4425a54e56a6812323f06d43ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027938c212a04dcdbcfca83bf772da16" is_pre_election: true
I20260923 14:40:04.431517 22549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27d9369022e4c81b0d521edb53bd1c4" candidate_uuid: "e2d9ed4425a54e56a6812323f06d43ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" is_pre_election: true
I20260923 14:40:04.431754 22415 raft_consensus.cc:2454] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d9ed4425a54e56a6812323f06d43ed in term 0.
I20260923 14:40:04.431762 22549 raft_consensus.cc:2454] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d9ed4425a54e56a6812323f06d43ed in term 0.
I20260923 14:40:04.432199 22216 leader_election.cc:302] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 027938c212a04dcdbcfca83bf772da16, c53e2d59dd814ce9af21a2ea2ee966da, e2d9ed4425a54e56a6812323f06d43ed; no voters:
I20260923 14:40:04.432435 22620 raft_consensus.cc:2781] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:40:04.432497 22620 raft_consensus.cc:482] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:40:04.432523 22620 raft_consensus.cc:3037] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:04.433440 22620 raft_consensus.cc:504] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 } } peers { permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 } } peers { permanent_uuid: "e2d9ed4425a54e56a6812323f06d43ed" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36885 } }
I20260923 14:40:04.433735 22620 leader_election.cc:288] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [CANDIDATE]: Term 1 election: Requested vote from peers 027938c212a04dcdbcfca83bf772da16 (127.2.218.2:40939), c53e2d59dd814ce9af21a2ea2ee966da (127.2.218.3:33049)
I20260923 14:40:04.434051 22549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27d9369022e4c81b0d521edb53bd1c4" candidate_uuid: "e2d9ed4425a54e56a6812323f06d43ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c53e2d59dd814ce9af21a2ea2ee966da"
I20260923 14:40:04.434127 22415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27d9369022e4c81b0d521edb53bd1c4" candidate_uuid: "e2d9ed4425a54e56a6812323f06d43ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "027938c212a04dcdbcfca83bf772da16"
I20260923 14:40:04.434183 22549 raft_consensus.cc:3037] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:04.434237 22415 raft_consensus.cc:3037] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:04.435058 22549 raft_consensus.cc:2454] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d9ed4425a54e56a6812323f06d43ed in term 1.
I20260923 14:40:04.435225 22415 raft_consensus.cc:2454] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d9ed4425a54e56a6812323f06d43ed in term 1.
I20260923 14:40:04.435341 22216 leader_election.cc:302] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c53e2d59dd814ce9af21a2ea2ee966da, e2d9ed4425a54e56a6812323f06d43ed; no voters:
I20260923 14:40:04.435482 22620 raft_consensus.cc:2781] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:04.435698 22620 raft_consensus.cc:686] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 1 LEADER]: Becoming Leader. State: Replica: e2d9ed4425a54e56a6812323f06d43ed, State: Running, Role: LEADER
I20260923 14:40:04.436028 22620 consensus_queue.cc:237] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 } } peers { permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 } } peers { permanent_uuid: "e2d9ed4425a54e56a6812323f06d43ed" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36885 } }
I20260923 14:40:04.438995 22136 catalog_manager.cc:5495] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed reported cstate change: term changed from 0 to 1, leader changed from <none> to e2d9ed4425a54e56a6812323f06d43ed (127.2.218.1). New cstate: current_term: 1 leader_uuid: "e2d9ed4425a54e56a6812323f06d43ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2d9ed4425a54e56a6812323f06d43ed" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 36885 } health_report { overall_health: HEALTHY } } }
W20260923 14:40:04.444316 22462 tablet.cc:2369] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:40:04.471503 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.468367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45036 (local address 127.2.218.1:36885)
0923 14:40:04.468476 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:04.468481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:04.468544 (+ 63us) server_negotiation.cc:407] Connection header received
0923 14:40:04.468698 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:04.468703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:04.468766 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:04.468862 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:04.469095 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.469527 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.470298 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.470489 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.470587 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.470854 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:04.471179 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:04.471290 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:04.471342 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 14:40:04.471388 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 14:40:04.476493 22415 raft_consensus.cc:1261] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16 [term 1 FOLLOWER]: Refusing update from remote peer e2d9ed4425a54e56a6812323f06d43ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:04.476500 22549 raft_consensus.cc:1261] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da [term 1 FOLLOWER]: Refusing update from remote peer e2d9ed4425a54e56a6812323f06d43ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:04.477080 22623 consensus_queue.cc:1035] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c53e2d59dd814ce9af21a2ea2ee966da" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 33049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:04.477243 22620 consensus_queue.cc:1035] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "027938c212a04dcdbcfca83bf772da16" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:04.484848 22628 mvcc.cc:205] Tried to move back new op lower bound from 7332554360730189824 to 7332554360572739584. Current Snapshot: MvccSnapshot[applied={T|T < 7332554360730189824}]
I20260923 14:40:04.490824 22136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:40:04.490959 22136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:40:04.492798 22136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0662792aa29949d59b9ae8bd18c417c0: Sending DeleteTablet for 3 replicas of tablet b27d9369022e4c81b0d521edb53bd1c4
I20260923 14:40:04.493396 22529 tablet_service.cc:1514] Processing DeleteTablet for tablet b27d9369022e4c81b0d521edb53bd1c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:04 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:57764
I20260923 14:40:04.493577 22395 tablet_service.cc:1514] Processing DeleteTablet for tablet b27d9369022e4c81b0d521edb53bd1c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:04 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:53422
I20260923 14:40:04.493876 22261 tablet_service.cc:1514] Processing DeleteTablet for tablet b27d9369022e4c81b0d521edb53bd1c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:04 UTC) from {username='kudu', principal='kudu/127.2.218.62@KRBTEST.COM'} at 127.0.0.1:59622
I20260923 14:40:04.493939 22635 tablet_replica.cc:330] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: stopping tablet replica
I20260923 14:40:04.494174 22635 raft_consensus.cc:2229] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:40:04.494405 22635 raft_consensus.cc:2258] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:40:04.494563 22637 tablet_replica.cc:330] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed: stopping tablet replica
I20260923 14:40:04.494817 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 22198
I20260923 14:40:04.494901 22637 raft_consensus.cc:2229] T b27d9369022e4c81b0d521edb53bd1c4 P e2d9ed4425a54e56a6812323f06d43ed [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:40:04.495373 22636 tablet_replica.cc:330] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: stopping tablet replica
I20260923 14:40:04.495575 22636 raft_consensus.cc:2229] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:40:04.495720 22635 ts_tablet_manager.cc:1905] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:40:04.495780 22636 raft_consensus.cc:2258] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:40:04.496912 22636 ts_tablet_manager.cc:1905] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:40:04.497996 22635 ts_tablet_manager.cc:1918] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:40:04.498088 22635 log.cc:1196] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/b27d9369022e4c81b0d521edb53bd1c4
I20260923 14:40:04.498386 22635 ts_tablet_manager.cc:1939] T b27d9369022e4c81b0d521edb53bd1c4 P c53e2d59dd814ce9af21a2ea2ee966da: Deleting consensus metadata
I20260923 14:40:04.499162 22636 ts_tablet_manager.cc:1918] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:40:04.499241 22636 log.cc:1196] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790174173335390-2920-0/minicluster-data/ts-1/wal/wals/b27d9369022e4c81b0d521edb53bd1c4
I20260923 14:40:04.499459 22636 ts_tablet_manager.cc:1939] T b27d9369022e4c81b0d521edb53bd1c4 P 027938c212a04dcdbcfca83bf772da16: Deleting consensus metadata
I20260923 14:40:04.499411 22122 catalog_manager.cc:4841] TS c53e2d59dd814ce9af21a2ea2ee966da (127.2.218.3:33049): tablet b27d9369022e4c81b0d521edb53bd1c4 (table test-table [id=66da8bd03dba4651bbc238411b353d5a]) successfully deleted
I20260923 14:40:04.500167 22123 catalog_manager.cc:4841] TS 027938c212a04dcdbcfca83bf772da16 (127.2.218.2:40939): tablet b27d9369022e4c81b0d521edb53bd1c4 (table test-table [id=66da8bd03dba4651bbc238411b353d5a]) successfully deleted
W20260923 14:40:04.501555 22123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:36885 (error 108)
I20260923 14:40:04.502242 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 22332
I20260923 14:40:04.502426 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.502168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:36885 (local address 127.0.0.1:45042)
0923 14:40:04.502286 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:04.502353 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:36885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 14:40:04.502928 22123 catalog_manager.cc:4584] TS e2d9ed4425a54e56a6812323f06d43ed (127.2.218.1:36885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b27d9369022e4c81b0d521edb53bd1c4: Network error: Client connection negotiation failed: client connection to 127.2.218.1:36885: connect: Connection refused (error 111)
I20260923 14:40:04.507169 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 22466
I20260923 14:40:04.512305 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 22104
2026-09-23T14:40:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2121 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T14:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:04Z Disabled control of system clock
I20260923 14:40:04.539529 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:34723
--webserver_interface=127.2.218.62
--webserver_port=0
--builtin_ntp_servers=127.2.218.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:34723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:40:04.629782 22642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:04.629959 22642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:04.629987 22642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:04.633255 22642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:04.633292 22642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:04.633306 22642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:04.633320 22642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:04.633333 22642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:04.637349 22642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:34723
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:34723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:04.638003 22642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:04.639004 22642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:04.645710 22647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:04.645701 22648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:04.645917 22650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:04.645964 22642 server_base.cc:1034] running on GCE node
I20260923 14:40:04.646503 22642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:04.647332 22642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:04.648487 22642 hybrid_clock.cc:648] HybridClock initialized: now 1790174404648471 us; error 25 us; skew 500 ppm
I20260923 14:40:04.650060 22642 webserver.cc:460] Webserver started at http://127.2.218.62:38085/ using document root <none> and password file <none>
I20260923 14:40:04.650627 22642 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:04.650684 22642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:04.650827 22642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:04.652413 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "a5ad315c683a4846ba8268b152a13371"
format_stamp: "Formatted at 2026-09-23 14:40:04 on dist-test-slave-zl47"
server_key: "c61549b1fbedb4c78375cf54f9c5e130"
server_key_iv: "bedbb1dab091525fa96e78a65bb485df"
server_key_version: "encryptionkey@0"
I20260923 14:40:04.652823 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "a5ad315c683a4846ba8268b152a13371"
format_stamp: "Formatted at 2026-09-23 14:40:04 on dist-test-slave-zl47"
server_key: "c61549b1fbedb4c78375cf54f9c5e130"
server_key_iv: "bedbb1dab091525fa96e78a65bb485df"
server_key_version: "encryptionkey@0"
I20260923 14:40:04.655711 22642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260923 14:40:04.657616 22656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:04.658740 22642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260923 14:40:04.658830 22642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "a5ad315c683a4846ba8268b152a13371"
format_stamp: "Formatted at 2026-09-23 14:40:04 on dist-test-slave-zl47"
server_key: "c61549b1fbedb4c78375cf54f9c5e130"
server_key_iv: "bedbb1dab091525fa96e78a65bb485df"
server_key_version: "encryptionkey@0"
I20260923 14:40:04.658908 22642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:04.670212 22642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:04.670665 22642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:04.670776 22642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:04.676270 22642 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:34723
I20260923 14:40:04.676280 22708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:34723 every 8 connection(s)
I20260923 14:40:04.676976 22642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:04.679373 22709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:04.683490 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 22642
I20260923 14:40:04.683596 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:04.683642 22709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371: Bootstrap starting.
I20260923 14:40:04.683863 2920 external_mini_cluster.cc:1428] Setting key ec3f639bd1c79eeda95fe57ed3efcb1a
I20260923 14:40:04.685798 22709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:04.686484 22709 log.cc:824] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:04.687996 22709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371: No bootstrap required, opened a new log
I20260923 14:40:04.691008 22709 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ad315c683a4846ba8268b152a13371" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34723 } }
I20260923 14:40:04.691200 22709 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:04.691272 22709 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ad315c683a4846ba8268b152a13371, State: Initialized, Role: FOLLOWER
I20260923 14:40:04.691686 22709 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ad315c683a4846ba8268b152a13371" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34723 } }
I20260923 14:40:04.691823 22709 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:04.691885 22709 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:04.691965 22709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:04.692523 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.685086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.2.218.62:34723)
0923 14:40:04.685619 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:04.685632 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:04.685661 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:40:04.686326 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:04.686344 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:04.686723 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:04.687007 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:04.687431 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.688440 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.689332 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.689801 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.689986 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.690809 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:04.690815 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:04.690825 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:04.691145 (+ 320us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:04.691194 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:04.691238 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:04.691330 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:04.691549 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 14:40:04.691823 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260923 14:40:04.692693 22709 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ad315c683a4846ba8268b152a13371" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34723 } }
I20260923 14:40:04.693009 22709 leader_election.cc:302] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5ad315c683a4846ba8268b152a13371; no voters:
I20260923 14:40:04.693284 22709 leader_election.cc:288] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:04.693347 22714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:04.693511 22714 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [term 1 LEADER]: Becoming Leader. State: Replica: a5ad315c683a4846ba8268b152a13371, State: Running, Role: LEADER
I20260923 14:40:04.693754 22714 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ad315c683a4846ba8268b152a13371" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34723 } }
I20260923 14:40:04.694345 22709 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:04.694757 22714 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5ad315c683a4846ba8268b152a13371. Latest consensus state: current_term: 1 leader_uuid: "a5ad315c683a4846ba8268b152a13371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ad315c683a4846ba8268b152a13371" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34723 } } }
I20260923 14:40:04.694846 22714 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:04.694802 22715 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5ad315c683a4846ba8268b152a13371" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ad315c683a4846ba8268b152a13371" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 34723 } } }
I20260923 14:40:04.694919 22715 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:04.695214 22718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:04.697808 22718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:04.702539 22718 catalog_manager.cc:1347] Generated new cluster ID: b53032fa0bfb4daaa84759691a8fecf8
I20260923 14:40:04.702597 22718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:04.723541 22718 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:04.724478 22718 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:04.734686 22718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371: Generated new TSK 0
I20260923 14:40:04.735245 22718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:04.742197 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.1:0
--local_ip_for_outbound_sockets=127.2.218.1
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34723
--builtin_ntp_servers=127.2.218.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 14:40:04.837103 22733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:04.837296 22733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:04.837358 22733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:04.840713 22733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:04.840760 22733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:04.840852 22733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.1
I20260923 14:40:04.845008 22733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.218.1
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.1
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:04.845763 22733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:04.846868 22733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:04.854672 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:04.854660 22741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:04.854691 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:04.854848 22733 server_base.cc:1034] running on GCE node
I20260923 14:40:04.855234 22733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:04.855747 22733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:04.856901 22733 hybrid_clock.cc:648] HybridClock initialized: now 1790174404856882 us; error 46 us; skew 500 ppm
I20260923 14:40:04.858551 22733 webserver.cc:460] Webserver started at http://127.2.218.1:42717/ using document root <none> and password file <none>
I20260923 14:40:04.859018 22733 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:04.859100 22733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:04.859295 22733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:04.860914 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/data/instance:
uuid: "945d505e6838440eb39955330f0819a6"
format_stamp: "Formatted at 2026-09-23 14:40:04 on dist-test-slave-zl47"
server_key: "06c1a0b40bdd38dcb5b02c288b48635a"
server_key_iv: "fedc9e2755b368ae1c787f7328e6462a"
server_key_version: "encryptionkey@0"
I20260923 14:40:04.861443 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance:
uuid: "945d505e6838440eb39955330f0819a6"
format_stamp: "Formatted at 2026-09-23 14:40:04 on dist-test-slave-zl47"
server_key: "06c1a0b40bdd38dcb5b02c288b48635a"
server_key_iv: "fedc9e2755b368ae1c787f7328e6462a"
server_key_version: "encryptionkey@0"
I20260923 14:40:04.864516 22733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:40:04.866503 22747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:04.867514 22733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:04.867627 22733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/wal
uuid: "945d505e6838440eb39955330f0819a6"
format_stamp: "Formatted at 2026-09-23 14:40:04 on dist-test-slave-zl47"
server_key: "06c1a0b40bdd38dcb5b02c288b48635a"
server_key_iv: "fedc9e2755b368ae1c787f7328e6462a"
server_key_version: "encryptionkey@0"
I20260923 14:40:04.867715 22733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:04.877691 22733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:04.878180 22733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:04.878332 22733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:04.878765 22733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:04.879468 22733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:04.879544 22733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:04.879601 22733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:04.879630 22733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:04.887151 22733 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.1:43133
I20260923 14:40:04.887198 22860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.1:43133 every 8 connection(s)
I20260923 14:40:04.887884 22733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/data/info.pb
I20260923 14:40:04.894999 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.889321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:37843 (local address 127.2.218.62:34723)
0923 14:40:04.889458 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:04.889462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:04.890369 (+ 907us) server_negotiation.cc:407] Connection header received
0923 14:40:04.891391 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:04.891393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:04.891433 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:04.891495 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:04.892159 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.892603 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.893502 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.893684 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.893751 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.894615 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:04.894618 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:04.894620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:04.894629 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:04.894660 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:04.894667 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:04.894717 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:04.894821 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 14:40:04.894873 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 14:40:04.895795 22863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:04.889694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:34723 (local address 127.2.218.1:37843)
0923 14:40:04.890251 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:04.890281 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:04.891241 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:04.891622 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:04.891627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:04.891692 (+ 65us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:04.892053 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.892063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.892697 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.892699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.893409 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:04.893417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:04.893790 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:04.893793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:04.893914 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:04.894393 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:40:04.894399 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:04.894430 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:40:04.894478 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:04.894719 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:04.894726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:04.894729 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:40:04.894806 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 14:40:04.895025 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
I20260923 14:40:04.896795 22861 heartbeater.cc:347] Connected to a master server at 127.2.218.62:34723
I20260923 14:40:04.896998 22861 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:04.897456 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 22733
I20260923 14:40:04.897493 22861 heartbeater.cc:511] Master 127.2.218.62:34723 requested a full tablet report, sending...
I20260923 14:40:04.897560 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-0/wal/instance
I20260923 14:40:04.897850 2920 external_mini_cluster.cc:1428] Setting key 2ceb8a9e21f712f69f9a0602a1624970
I20260923 14:40:04.898838 22673 ts_manager.cc:194] Registered new tserver with Master: 945d505e6838440eb39955330f0819a6 (127.2.218.1:43133)
I20260923 14:40:04.899636 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.2:0
--local_ip_for_outbound_sockets=127.2.218.2
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34723
--builtin_ntp_servers=127.2.218.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:40:04.899854 22673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.1:37843
W20260923 14:40:04.996397 22864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:04.996587 22864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:04.996620 22864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:04.999881 22864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:04.999920 22864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:04.999966 22864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.2
I20260923 14:40:05.003767 22864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.218.2
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.2
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:05.004484 22864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:05.005487 22864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:40:05.013267 22864 server_base.cc:1034] running on GCE node
W20260923 14:40:05.013236 22870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:05.013386 22869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:05.013237 22872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:05.013739 22864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:05.014293 22864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:05.015465 22864 hybrid_clock.cc:648] HybridClock initialized: now 1790174405015454 us; error 22 us; skew 500 ppm
I20260923 14:40:05.017051 22864 webserver.cc:460] Webserver started at http://127.2.218.2:42263/ using document root <none> and password file <none>
I20260923 14:40:05.017498 22864 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:05.017575 22864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:05.017762 22864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:05.019451 22864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/data/instance:
uuid: "255a8c0aa2ee43cd91d19e32f3c18495"
format_stamp: "Formatted at 2026-09-23 14:40:05 on dist-test-slave-zl47"
server_key: "790cea28f500d1a31da3df266864f019"
server_key_iv: "893f2ddb565bf9672d8662a86cd6526b"
server_key_version: "encryptionkey@0"
I20260923 14:40:05.019937 22864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance:
uuid: "255a8c0aa2ee43cd91d19e32f3c18495"
format_stamp: "Formatted at 2026-09-23 14:40:05 on dist-test-slave-zl47"
server_key: "790cea28f500d1a31da3df266864f019"
server_key_iv: "893f2ddb565bf9672d8662a86cd6526b"
server_key_version: "encryptionkey@0"
I20260923 14:40:05.022861 22864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260923 14:40:05.024803 22878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:05.025746 22864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:05.025876 22864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/wal
uuid: "255a8c0aa2ee43cd91d19e32f3c18495"
format_stamp: "Formatted at 2026-09-23 14:40:05 on dist-test-slave-zl47"
server_key: "790cea28f500d1a31da3df266864f019"
server_key_iv: "893f2ddb565bf9672d8662a86cd6526b"
server_key_version: "encryptionkey@0"
I20260923 14:40:05.025972 22864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:05.047326 22864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:05.047809 22864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:05.047972 22864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:05.048416 22864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:05.049139 22864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:05.049201 22864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:05.049258 22864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:05.049300 22864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:05.056954 22864 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.2:40389
I20260923 14:40:05.056988 22991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.2:40389 every 8 connection(s)
I20260923 14:40:05.057662 22864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/data/info.pb
I20260923 14:40:05.065400 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.059181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.2:42347 (local address 127.2.218.62:34723)
0923 14:40:05.059311 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:05.059315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:05.060170 (+ 855us) server_negotiation.cc:407] Connection header received
0923 14:40:05.061285 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:05.061306 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:05.061375 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:05.061447 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:05.062166 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.062671 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.063838 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.064046 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.064131 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.064969 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:05.064972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:05.064973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:05.064983 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:05.064997 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:05.065004 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:05.065053 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:05.065166 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 14:40:05.065251 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:40:05.065402 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 22864
I20260923 14:40:05.065501 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-1/wal/instance
I20260923 14:40:05.065717 2920 external_mini_cluster.cc:1428] Setting key 5326c002df2afb893789f50c424eda33
I20260923 14:40:05.066038 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.059510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:34723 (local address 127.2.218.2:42347)
0923 14:40:05.060048 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:05.060083 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:05.061125 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:05.061560 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:05.061565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:05.061634 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:05.062008 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.062019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.062789 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.062792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.063566 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.063576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.064168 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.064171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.064288 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.064760 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:40:05.064761 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:05.064803 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:40:05.064852 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:05.065067 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:05.065073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:05.065076 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:40:05.065152 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 14:40:05.065324 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":89,"threads_started":1}
I20260923 14:40:05.067040 22992 heartbeater.cc:347] Connected to a master server at 127.2.218.62:34723
I20260923 14:40:05.067224 22992 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:05.067507 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.218.3:0
--local_ip_for_outbound_sockets=127.2.218.3
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34723
--builtin_ntp_servers=127.2.218.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 14:40:05.067698 22992 heartbeater.cc:511] Master 127.2.218.62:34723 requested a full tablet report, sending...
I20260923 14:40:05.068678 22673 ts_manager.cc:194] Registered new tserver with Master: 255a8c0aa2ee43cd91d19e32f3c18495 (127.2.218.2:40389)
I20260923 14:40:05.069191 22673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.2:42347
W20260923 14:40:05.171307 22995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:05.171657 22995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:05.171727 22995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:05.178066 22995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:05.178157 22995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:05.178244 22995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.218.3
I20260923 14:40:05.184741 22995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.218.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.218.3
--webserver_port=0
--tserver_master_addrs=127.2.218.62:34723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.218.3
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:05.185566 22995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:05.186829 22995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:05.196038 23000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:05.196048 23001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:05.196332 23003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:05.196476 22995 server_base.cc:1034] running on GCE node
I20260923 14:40:05.196830 22995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:05.197315 22995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:05.198496 22995 hybrid_clock.cc:648] HybridClock initialized: now 1790174405198452 us; error 56 us; skew 500 ppm
I20260923 14:40:05.199922 22995 webserver.cc:460] Webserver started at http://127.2.218.3:39647/ using document root <none> and password file <none>
I20260923 14:40:05.200436 22995 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:05.200490 22995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:05.200690 22995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:05.202414 22995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/data/instance:
uuid: "12c4342a8c1b4c239aa057cf1874474f"
format_stamp: "Formatted at 2026-09-23 14:40:05 on dist-test-slave-zl47"
server_key: "3968234bf61a2d6be5512371405f7ab7"
server_key_iv: "3ae1affe681283a946f27311e7f7de39"
server_key_version: "encryptionkey@0"
I20260923 14:40:05.202916 22995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance:
uuid: "12c4342a8c1b4c239aa057cf1874474f"
format_stamp: "Formatted at 2026-09-23 14:40:05 on dist-test-slave-zl47"
server_key: "3968234bf61a2d6be5512371405f7ab7"
server_key_iv: "3ae1affe681283a946f27311e7f7de39"
server_key_version: "encryptionkey@0"
I20260923 14:40:05.205763 22995 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:40:05.207720 23009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:05.208658 22995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:40:05.208773 22995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/wal
uuid: "12c4342a8c1b4c239aa057cf1874474f"
format_stamp: "Formatted at 2026-09-23 14:40:05 on dist-test-slave-zl47"
server_key: "3968234bf61a2d6be5512371405f7ab7"
server_key_iv: "3ae1affe681283a946f27311e7f7de39"
server_key_version: "encryptionkey@0"
I20260923 14:40:05.208866 22995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:05.220939 22995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:05.221407 22995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:05.221556 22995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:05.222019 22995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 14:40:05.222780 22995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 14:40:05.222849 22995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:05.222927 22995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 14:40:05.222972 22995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:05.230641 22995 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.3:39245
I20260923 14:40:05.230675 23122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.3:39245 every 8 connection(s)
I20260923 14:40:05.231318 22995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/data/info.pb
I20260923 14:40:05.233217 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 22995
I20260923 14:40:05.233338 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/wal/instance
I20260923 14:40:05.233676 2920 external_mini_cluster.cc:1428] Setting key 13420961dc300741cf7b095b6a75509d
I20260923 14:40:05.239269 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.232820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.3:52517 (local address 127.2.218.62:34723)
0923 14:40:05.232970 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:05.232975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:05.233818 (+ 843us) server_negotiation.cc:407] Connection header received
0923 14:40:05.235297 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:05.235300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:05.235350 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:05.235438 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:05.236290 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.236767 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.237741 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.237932 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.238005 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.238890 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:05.238892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:05.238894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:05.238906 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:05.238928 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:05.238937 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:05.239006 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:05.239123 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 14:40:05.239171 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:40:05.239892 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.233098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:34723 (local address 127.2.218.3:52517)
0923 14:40:05.233697 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:05.233738 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:05.235104 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:05.235561 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:05.235566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:05.235664 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:05.236174 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.236185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.236871 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.236874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.237650 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.237658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.238040 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.238042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.238206 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.238676 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 14:40:05.238678 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:05.238710 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 14:40:05.238758 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:05.239016 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:05.239021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:05.239024 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 14:40:05.239102 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 14:40:05.239270 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":120,"threads_started":1}
I20260923 14:40:05.240792 23123 heartbeater.cc:347] Connected to a master server at 127.2.218.62:34723
I20260923 14:40:05.241014 23123 heartbeater.cc:464] Registering TS with master...
I20260923 14:40:05.241418 23123 heartbeater.cc:511] Master 127.2.218.62:34723 requested a full tablet report, sending...
I20260923 14:40:05.242264 22673 ts_manager.cc:194] Registered new tserver with Master: 12c4342a8c1b4c239aa057cf1874474f (127.2.218.3:39245)
I20260923 14:40:05.242715 22673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.218.3:52517
I20260923 14:40:05.247311 2920 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 14:40:05.252002 22712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.248915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.2.218.62:34723)
0923 14:40:05.249028 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:05.249032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:05.249154 (+ 122us) server_negotiation.cc:407] Connection header received
0923 14:40:05.249235 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:05.249236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:05.249269 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:05.249329 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:05.249530 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.249960 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.250711 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.250934 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.251012 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.251552 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:05.251554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:05.251556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:05.251566 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:05.251581 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:05.251588 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:05.251645 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:05.251838 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 14:40:05.251899 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:40:05.254285 22673 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 14:40:05.256765 22673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 14:40:05.269023 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.263897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:39245 (local address 127.0.0.1:56624)
0923 14:40:05.264217 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:05.264267 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:05.264401 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:05.265078 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:05.265082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:05.265112 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:05.265258 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.265266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.266486 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.266489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.267508 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.267517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.268571 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.268574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.268664 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.268693 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 14:40:05.268781 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 14:40:05.268893 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
I20260923 14:40:05.269256 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.264488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:43133 (local address 127.0.0.1:41088)
0923 14:40:05.265159 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:05.265169 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:05.265258 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:05.265875 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:05.265878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:05.265895 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:05.265986 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.265994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.267184 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.267187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.268119 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.268128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.268883 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.268887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.268970 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.268994 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 14:40:05.269060 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 14:40:05.269127 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":81,"threads_started":1}
I20260923 14:40:05.269340 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.264774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.2.218.1:43133)
0923 14:40:05.265484 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:05.265495 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:05.265514 (+ 19us) server_negotiation.cc:407] Connection header received
0923 14:40:05.265577 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:05.265581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:05.265730 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:05.265908 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:05.266116 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.267088 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.268249 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.268736 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.268875 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.269042 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:05.269092 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 14:40:05.269190 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":128,"threads_started":1}
I20260923 14:40:05.269536 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.263970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.2.218.3:39245)
0923 14:40:05.264357 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:05.264369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:05.264393 (+ 24us) server_negotiation.cc:407] Connection header received
0923 14:40:05.264585 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:05.264590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:05.264763 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:05.264916 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:05.265447 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.266374 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.267633 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.268257 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.268355 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.268516 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:05.269294 (+ 778us) server_negotiation.cc:299] Negotiation successful
0923 14:40:05.269391 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":95,"threads_started":1}
I20260923 14:40:05.270148 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.264323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40389 (local address 127.0.0.1:52052)
0923 14:40:05.265085 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:05.265108 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:05.265187 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:05.266039 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:05.266042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:05.266061 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:05.266197 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.266202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.267052 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.267056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.268092 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.268102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.269281 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.269284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.269772 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.269792 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:40:05.269909 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 14:40:05.269970 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":75,"threads_started":1}
I20260923 14:40:05.270355 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.264658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52052 (local address 127.2.218.2:40389)
0923 14:40:05.265539 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:05.265547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:05.265565 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:40:05.265621 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:05.265624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:05.265742 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:05.265898 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:05.266308 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.266934 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.268219 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.268964 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.269508 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.269675 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:05.269916 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 14:40:05.270007 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":146,"threads_started":1}
I20260923 14:40:05.271931 23057 tablet_service.cc:1467] Processing CreateTablet for tablet 2cea2cb59a694a05a6686da6304d6a4d (DEFAULT_TABLE table=test-table [id=ca3317277df648c586042810c35adcb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:05.272709 23057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cea2cb59a694a05a6686da6304d6a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:05.272459 22926 tablet_service.cc:1467] Processing CreateTablet for tablet 2cea2cb59a694a05a6686da6304d6a4d (DEFAULT_TABLE table=test-table [id=ca3317277df648c586042810c35adcb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:05.273212 22926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cea2cb59a694a05a6686da6304d6a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:05.275686 22795 tablet_service.cc:1467] Processing CreateTablet for tablet 2cea2cb59a694a05a6686da6304d6a4d (DEFAULT_TABLE table=test-table [id=ca3317277df648c586042810c35adcb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 14:40:05.276561 22795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cea2cb59a694a05a6686da6304d6a4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:05.278147 23140 tablet_bootstrap.cc:492] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: Bootstrap starting.
I20260923 14:40:05.280187 23140 tablet_bootstrap.cc:654] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:05.280817 23140 log.cc:824] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:05.281375 23141 tablet_bootstrap.cc:492] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495: Bootstrap starting.
I20260923 14:40:05.282295 23143 tablet_bootstrap.cc:492] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6: Bootstrap starting.
I20260923 14:40:05.282526 23140 tablet_bootstrap.cc:492] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: No bootstrap required, opened a new log
I20260923 14:40:05.282686 23140 ts_tablet_manager.cc:1397] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:40:05.283143 23141 tablet_bootstrap.cc:654] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:05.283861 23141 log.cc:824] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:05.284243 23143 tablet_bootstrap.cc:654] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:05.284978 23143 log.cc:824] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:05.285594 23141 tablet_bootstrap.cc:492] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495: No bootstrap required, opened a new log
I20260923 14:40:05.285775 23141 ts_tablet_manager.cc:1397] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:40:05.286019 23140 raft_consensus.cc:348] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 } } peers { permanent_uuid: "945d505e6838440eb39955330f0819a6" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43133 } } peers { permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 } }
I20260923 14:40:05.286266 23140 raft_consensus.cc:374] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:05.286312 23140 raft_consensus.cc:729] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c4342a8c1b4c239aa057cf1874474f, State: Initialized, Role: FOLLOWER
I20260923 14:40:05.286688 23140 consensus_queue.cc:260] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 } } peers { permanent_uuid: "945d505e6838440eb39955330f0819a6" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43133 } } peers { permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 } }
I20260923 14:40:05.286751 23143 tablet_bootstrap.cc:492] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6: No bootstrap required, opened a new log
I20260923 14:40:05.286906 23143 ts_tablet_manager.cc:1397] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:40:05.287204 23123 heartbeater.cc:503] Master 127.2.218.62:34723 was elected leader, sending a full tablet report...
I20260923 14:40:05.288210 23140 ts_tablet_manager.cc:1428] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260923 14:40:05.289503 23141 raft_consensus.cc:348] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 } } peers { permanent_uuid: "945d505e6838440eb39955330f0819a6" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43133 } } peers { permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 } }
I20260923 14:40:05.289717 23141 raft_consensus.cc:374] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:05.289772 23141 raft_consensus.cc:729] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 255a8c0aa2ee43cd91d19e32f3c18495, State: Initialized, Role: FOLLOWER
I20260923 14:40:05.290162 23141 consensus_queue.cc:260] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 } } peers { permanent_uuid: "945d505e6838440eb39955330f0819a6" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43133 } } peers { permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 } }
I20260923 14:40:05.290750 22992 heartbeater.cc:503] Master 127.2.218.62:34723 was elected leader, sending a full tablet report...
I20260923 14:40:05.290982 23141 ts_tablet_manager.cc:1428] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260923 14:40:05.290592 23143 raft_consensus.cc:348] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 } } peers { permanent_uuid: "945d505e6838440eb39955330f0819a6" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43133 } } peers { permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 } }
I20260923 14:40:05.291683 23143 raft_consensus.cc:374] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:05.291747 23143 raft_consensus.cc:729] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945d505e6838440eb39955330f0819a6, State: Initialized, Role: FOLLOWER
I20260923 14:40:05.292111 23143 consensus_queue.cc:260] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 } } peers { permanent_uuid: "945d505e6838440eb39955330f0819a6" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43133 } } peers { permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 } }
I20260923 14:40:05.292693 22861 heartbeater.cc:503] Master 127.2.218.62:34723 was elected leader, sending a full tablet report...
I20260923 14:40:05.292946 23143 ts_tablet_manager.cc:1428] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260923 14:40:05.308439 22993 tablet.cc:2369] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 14:40:05.388866 22862 tablet.cc:2369] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:40:05.471592 23148 raft_consensus.cc:482] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 14:40:05.471755 23148 raft_consensus.cc:504] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 } } peers { permanent_uuid: "945d505e6838440eb39955330f0819a6" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43133 } } peers { permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 } }
I20260923 14:40:05.472754 23148 leader_election.cc:288] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12c4342a8c1b4c239aa057cf1874474f (127.2.218.3:39245), 255a8c0aa2ee43cd91d19e32f3c18495 (127.2.218.2:40389)
I20260923 14:40:05.476258 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.472938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40389 (local address 127.2.218.1:55561)
0923 14:40:05.473229 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:05.473239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:05.473315 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:05.473553 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:05.473556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:05.473570 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:05.473695 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.473701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.474512 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.474516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.475249 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.475257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.475886 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.475891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.475978 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.475998 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 14:40:05.476062 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 14:40:05.476108 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260923 14:40:05.476300 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.473082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:55561 (local address 127.2.218.2:40389)
0923 14:40:05.473200 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:05.473204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:05.473309 (+ 105us) server_negotiation.cc:407] Connection header received
0923 14:40:05.473416 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:05.473418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:05.473468 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:05.473543 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:05.473806 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.474403 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.475375 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.475766 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.475854 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.475951 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:05.476101 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 14:40:05.476183 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:40:05.476380 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.473027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.218.1:34715 (local address 127.2.218.3:39245)
0923 14:40:05.473160 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:05.473164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:05.473426 (+ 262us) server_negotiation.cc:407] Connection header received
0923 14:40:05.473546 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:05.473548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:05.473590 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:05.473656 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 14:40:05.473925 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.474423 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.475452 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.475805 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.475943 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.475991 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:05.476208 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 14:40:05.476272 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 14:40:05.476437 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.472943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.3:39245 (local address 127.2.218.1:34715)
0923 14:40:05.473318 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:05.473333 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:05.473433 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:05.473684 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:05.473686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:05.473699 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 14:40:05.473818 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.473825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.474558 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.474560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.475295 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:05.475329 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.475962 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.475968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:05.476083 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.476108 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 14:40:05.476183 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 14:40:05.476241 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":25,"spinlock_wait_cycles":28544,"thread_start_us":112,"threads_started":1}
I20260923 14:40:05.476845 22946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cea2cb59a694a05a6686da6304d6a4d" candidate_uuid: "945d505e6838440eb39955330f0819a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" is_pre_election: true
I20260923 14:40:05.477094 22946 raft_consensus.cc:2454] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 945d505e6838440eb39955330f0819a6 in term 0.
I20260923 14:40:05.477412 23077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cea2cb59a694a05a6686da6304d6a4d" candidate_uuid: "945d505e6838440eb39955330f0819a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c4342a8c1b4c239aa057cf1874474f" is_pre_election: true
I20260923 14:40:05.477548 22751 leader_election.cc:302] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 255a8c0aa2ee43cd91d19e32f3c18495, 945d505e6838440eb39955330f0819a6; no voters:
I20260923 14:40:05.477685 23077 raft_consensus.cc:2454] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 945d505e6838440eb39955330f0819a6 in term 0.
I20260923 14:40:05.477793 23148 raft_consensus.cc:2781] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 14:40:05.477866 23148 raft_consensus.cc:482] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 14:40:05.477941 23148 raft_consensus.cc:3037] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:05.478852 23148 raft_consensus.cc:504] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 } } peers { permanent_uuid: "945d505e6838440eb39955330f0819a6" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43133 } } peers { permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 } }
I20260923 14:40:05.479174 23148 leader_election.cc:288] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [CANDIDATE]: Term 1 election: Requested vote from peers 12c4342a8c1b4c239aa057cf1874474f (127.2.218.3:39245), 255a8c0aa2ee43cd91d19e32f3c18495 (127.2.218.2:40389)
I20260923 14:40:05.479439 23077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cea2cb59a694a05a6686da6304d6a4d" candidate_uuid: "945d505e6838440eb39955330f0819a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c4342a8c1b4c239aa057cf1874474f"
I20260923 14:40:05.479460 22946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cea2cb59a694a05a6686da6304d6a4d" candidate_uuid: "945d505e6838440eb39955330f0819a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "255a8c0aa2ee43cd91d19e32f3c18495"
I20260923 14:40:05.479565 23077 raft_consensus.cc:3037] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:05.479593 22946 raft_consensus.cc:3037] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:05.480458 22946 raft_consensus.cc:2454] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 945d505e6838440eb39955330f0819a6 in term 1.
I20260923 14:40:05.480553 23077 raft_consensus.cc:2454] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 945d505e6838440eb39955330f0819a6 in term 1.
I20260923 14:40:05.480695 22751 leader_election.cc:302] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 255a8c0aa2ee43cd91d19e32f3c18495, 945d505e6838440eb39955330f0819a6; no voters:
I20260923 14:40:05.480916 23148 raft_consensus.cc:2781] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:05.481087 23148 raft_consensus.cc:686] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 1 LEADER]: Becoming Leader. State: Replica: 945d505e6838440eb39955330f0819a6, State: Running, Role: LEADER
I20260923 14:40:05.481374 23148 consensus_queue.cc:237] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 } } peers { permanent_uuid: "945d505e6838440eb39955330f0819a6" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43133 } } peers { permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 } }
W20260923 14:40:05.482049 23124 tablet.cc:2369] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:40:05.484201 22672 catalog_manager.cc:5495] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 945d505e6838440eb39955330f0819a6 (127.2.218.1). New cstate: current_term: 1 leader_uuid: "945d505e6838440eb39955330f0819a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "945d505e6838440eb39955330f0819a6" member_type: VOTER last_known_addr { host: "127.2.218.1" port: 43133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 } health_report { overall_health: UNKNOWN } } }
I20260923 14:40:05.497750 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.494747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.2.218.1:43133)
0923 14:40:05.494851 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:05.494878 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:05.494894 (+ 16us) server_negotiation.cc:407] Connection header received
0923 14:40:05.495038 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:05.495041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:05.495082 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:05.495181 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:05.495434 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.495840 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.496574 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:05.496760 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:05.496852 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:05.497115 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:05.497431 (+ 316us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:05.497542 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:05.497600 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 14:40:05.497640 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 14:40:05.502226 22946 raft_consensus.cc:1261] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495 [term 1 FOLLOWER]: Refusing update from remote peer 945d505e6838440eb39955330f0819a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:05.502226 23077 raft_consensus.cc:1261] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f [term 1 FOLLOWER]: Refusing update from remote peer 945d505e6838440eb39955330f0819a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:05.502799 23151 consensus_queue.cc:1035] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "255a8c0aa2ee43cd91d19e32f3c18495" member_type: VOTER last_known_addr { host: "127.2.218.2" port: 40389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:05.502951 23148 consensus_queue.cc:1035] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c4342a8c1b4c239aa057cf1874474f" member_type: VOTER last_known_addr { host: "127.2.218.3" port: 39245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:05.510762 23156 mvcc.cc:205] Tried to move back new op lower bound from 7332554364931796992 to 7332554364853977088. Current Snapshot: MvccSnapshot[applied={T|T < 7332554364931796992}]
I20260923 14:40:05.515209 22673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:40:05.515332 22673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 14:40:05.517217 22673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5ad315c683a4846ba8268b152a13371: Sending DeleteTablet for 3 replicas of tablet 2cea2cb59a694a05a6686da6304d6a4d
I20260923 14:40:05.517845 23057 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cea2cb59a694a05a6686da6304d6a4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:05 UTC) from {username='slave'} at 127.0.0.1:56624
I20260923 14:40:05.517845 22795 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cea2cb59a694a05a6686da6304d6a4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:05 UTC) from {username='slave'} at 127.0.0.1:41088
I20260923 14:40:05.518244 22926 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cea2cb59a694a05a6686da6304d6a4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 14:40:05 UTC) from {username='slave'} at 127.0.0.1:52052
I20260923 14:40:05.518625 23166 tablet_replica.cc:330] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495: stopping tablet replica
I20260923 14:40:05.518749 23165 tablet_replica.cc:330] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6: stopping tablet replica
I20260923 14:40:05.518864 23166 raft_consensus.cc:2229] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:40:05.518924 23165 raft_consensus.cc:2229] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 1 LEADER]: Raft consensus shutting down.
I20260923 14:40:05.519088 23166 raft_consensus.cc:2258] T 2cea2cb59a694a05a6686da6304d6a4d P 255a8c0aa2ee43cd91d19e32f3c18495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:40:05.519147 23165 raft_consensus.cc:2258] T 2cea2cb59a694a05a6686da6304d6a4d P 945d505e6838440eb39955330f0819a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:40:05.519253 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 22733
I20260923 14:40:05.519385 23164 tablet_replica.cc:330] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: stopping tablet replica
I20260923 14:40:05.519863 23164 raft_consensus.cc:2229] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 14:40:05.520085 23164 raft_consensus.cc:2258] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 14:40:05.521313 23164 ts_tablet_manager.cc:1905] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 14:40:05.523161 23164 ts_tablet_manager.cc:1918] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 14:40:05.523227 23164 log.cc:1196] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: Deleting WAL directory at /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790174173335390-2920-0/minicluster-data/ts-2/wal/wals/2cea2cb59a694a05a6686da6304d6a4d
I20260923 14:40:05.523474 23164 ts_tablet_manager.cc:1939] T 2cea2cb59a694a05a6686da6304d6a4d P 12c4342a8c1b4c239aa057cf1874474f: Deleting consensus metadata
I20260923 14:40:05.524317 22657 catalog_manager.cc:4841] TS 12c4342a8c1b4c239aa057cf1874474f (127.2.218.3:39245): tablet 2cea2cb59a694a05a6686da6304d6a4d (table test-table [id=ca3317277df648c586042810c35adcb4]) successfully deleted
W20260923 14:40:05.525488 22657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.218.1:43133 (error 108)
I20260923 14:40:05.525631 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 22864
I20260923 14:40:05.526955 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.526675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.1:43133 (local address 127.0.0.1:41102)
0923 14:40:05.526810 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:05.526876 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.1:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 14:40:05.527164 22657 catalog_manager.cc:4584] TS 945d505e6838440eb39955330f0819a6 (127.2.218.1:43133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cea2cb59a694a05a6686da6304d6a4d: Network error: Client connection negotiation failed: client connection to 127.2.218.1:43133: connect: Connection refused (error 111)
I20260923 14:40:05.531167 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 22995
I20260923 14:40:05.532459 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:05.531216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.2:40389 (local address 127.0.0.1:52054)
0923 14:40:05.532330 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:05.532384 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.2:40389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1037}
I20260923 14:40:05.535809 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 22642
2026-09-23T14:40:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1022 ms)
[----------] 6 tests from EncryptionPolicyTest (9267 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:05 dist-test-slave-zl47 krb5kdc[23170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:05 dist-test-slave-zl47 krb5kdc[23170](info): set up 2 sockets
Sep 23 14:40:05 dist-test-slave-zl47 krb5kdc[23170](info): commencing operation
krb5kdc: starting...
W20260923 14:40:06.450349 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.895s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:06 dist-test-slave-zl47 krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:06.549427 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:06.644170 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:06.644390 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:06.644449 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:06.647854 23183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:06.647953 23183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:06.647985 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:06.648034 23183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:06.648063 23183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:06.652385 23183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:06.653121 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:06.653851 23183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:06.661134 23188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:06.661167 23183 server_base.cc:1034] running on GCE node
W20260923 14:40:06.661147 23189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:06.661320 23191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:06.661835 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:06.662817 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:06.663967 23183 hybrid_clock.cc:648] HybridClock initialized: now 1790174406663953 us; error 27 us; skew 500 ppm
Sep 23 14:40:06 dist-test-slave-zl47 krb5kdc[23170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:06.666560 23183 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:06.667356 23183 webserver.cc:460] Webserver started at http://127.0.0.1:44557/ using document root <none> and password file <none>
I20260923 14:40:06.667832 23183 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:06.667918 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:06.668094 23183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:06.669749 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "473052ebf7f6471896b6736e67b8c3e9"
format_stamp: "Formatted at 2026-09-23 14:40:06 on dist-test-slave-zl47"
server_key: "0c523197d12d68a9b93cafb1ee40785c"
server_key_iv: "56ac8396e37d8704ca14dfaa64b07f64"
server_key_version: "encryptionkey@0"
I20260923 14:40:06.670267 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "473052ebf7f6471896b6736e67b8c3e9"
format_stamp: "Formatted at 2026-09-23 14:40:06 on dist-test-slave-zl47"
server_key: "0c523197d12d68a9b93cafb1ee40785c"
server_key_iv: "56ac8396e37d8704ca14dfaa64b07f64"
server_key_version: "encryptionkey@0"
I20260923 14:40:06.673362 23183 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:40:06.675427 23198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:06.676389 23183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:40:06.676507 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "473052ebf7f6471896b6736e67b8c3e9"
format_stamp: "Formatted at 2026-09-23 14:40:06 on dist-test-slave-zl47"
server_key: "0c523197d12d68a9b93cafb1ee40785c"
server_key_iv: "56ac8396e37d8704ca14dfaa64b07f64"
server_key_version: "encryptionkey@0"
I20260923 14:40:06.676599 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:06.694497 23183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:06.696797 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:06.696960 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:06.703014 23183 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46315
I20260923 14:40:06.703038 23250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46315 every 8 connection(s)
I20260923 14:40:06.703742 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:06.706084 23251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:06.710968 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9: Bootstrap starting.
I20260923 14:40:06.712944 23251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:06.713573 23251 log.cc:824] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:06.713650 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 23183
I20260923 14:40:06.713802 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:06.714051 2920 external_mini_cluster.cc:1428] Setting key 26781bbdfb0742839316859bc46a5276
I20260923 14:40:06.715377 23251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9: No bootstrap required, opened a new log
I20260923 14:40:06.718199 23251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473052ebf7f6471896b6736e67b8c3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260923 14:40:06.718397 23251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:06.718465 23251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473052ebf7f6471896b6736e67b8c3e9, State: Initialized, Role: FOLLOWER
I20260923 14:40:06.718878 23251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473052ebf7f6471896b6736e67b8c3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260923 14:40:06.719009 23251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:06.719085 23251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:06.719187 23251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:06.719913 23251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473052ebf7f6471896b6736e67b8c3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260923 14:40:06.720242 23251 leader_election.cc:302] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 473052ebf7f6471896b6736e67b8c3e9; no voters:
I20260923 14:40:06.720535 23251 leader_election.cc:288] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:06.720592 23256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:06.720767 23256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [term 1 LEADER]: Becoming Leader. State: Replica: 473052ebf7f6471896b6736e67b8c3e9, State: Running, Role: LEADER
I20260923 14:40:06.721009 23256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473052ebf7f6471896b6736e67b8c3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } }
I20260923 14:40:06.721639 23251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [sys.catalog]: configured and running, proceeding with master startup.
Sep 23 14:40:06 dist-test-slave-zl47 krb5kdc[23170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 14:40:06.722122 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "473052ebf7f6471896b6736e67b8c3e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473052ebf7f6471896b6736e67b8c3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } } }
I20260923 14:40:06.722244 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:06.722572 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 473052ebf7f6471896b6736e67b8c3e9. Latest consensus state: current_term: 1 leader_uuid: "473052ebf7f6471896b6736e67b8c3e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473052ebf7f6471896b6736e67b8c3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46315 } } }
I20260923 14:40:06.722697 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:06.723316 23260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:06.726907 23260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:06.727234 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:06.714995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50177 (local address 127.0.0.1:46315)
0923 14:40:06.715330 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:06.715343 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:06.715369 (+ 26us) server_negotiation.cc:407] Connection header received
0923 14:40:06.716004 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:06.716019 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:06.716277 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:06.716549 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:06.717251 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:06.718035 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:06.718991 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:06.719533 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:06.719674 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:06.722510 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:06.722515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:06.722529 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:06.722554 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:06.724381 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:06.724796 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:06.724801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:06.724806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:06.724933 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:06.725261 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:06.725266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:06.725270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:06.725651 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:06.725832 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:06.726132 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 14:40:06.726315 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
I20260923 14:40:06.729962 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:06.731978 23260 catalog_manager.cc:1347] Generated new cluster ID: 2ffa5c4a4ce04aa78ec9ec7ba9b4a35c
I20260923 14:40:06.732029 23260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:06.742081 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:06.731820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58185 (local address 127.0.0.1:46315)
0923 14:40:06.731897 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:06.731901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:06.734602 (+ 2701us) server_negotiation.cc:407] Connection header received
0923 14:40:06.734762 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:06.734765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:06.734802 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:06.734877 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:06.735592 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:06.736303 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:06.737276 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:06.737546 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:06.737618 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:06.738921 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:06.738924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:06.738926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:06.738953 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:06.740344 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:06.740830 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:06.740869 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:06.740876 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:06.740953 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:06.741290 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:06.741293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:06.741294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:06.741466 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:06.741597 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:06.741844 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 14:40:06.741935 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 14:40:06.755509 23260 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:06.756265 23260 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:06.759848 23260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 473052ebf7f6471896b6736e67b8c3e9: Generated new TSK 0
I20260923 14:40:06.760470 23260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:06.765426 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 23183
W20260923 14:40:06.770247 23168 connection.cc:535] client connection to 127.0.0.1:46315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46315): Connection reset by peer (error 104)
2026-09-23T14:40:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:06 dist-test-slave-zl47 krb5kdc[23282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:06 dist-test-slave-zl47 krb5kdc[23282](info): set up 2 sockets
Sep 23 14:40:06 dist-test-slave-zl47 krb5kdc[23282](info): commencing operation
krb5kdc: starting...
W20260923 14:40:07.658075 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.874s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:07 dist-test-slave-zl47 krb5kdc[23282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:07.761760 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:07.854681 23295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:07.854872 23295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:07.854912 23295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:07.858490 23295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:07.858533 23295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:07.858549 23295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:07.858563 23295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:07.858601 23295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:07.862978 23295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:07.863739 23295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:07.864521 23295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:07.871227 23301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:07.871347 23295 server_base.cc:1034] running on GCE node
W20260923 14:40:07.871234 23300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:07.871214 23303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:07.871932 23295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:07.872752 23295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:07.873920 23295 hybrid_clock.cc:648] HybridClock initialized: now 1790174407873899 us; error 34 us; skew 500 ppm
Sep 23 14:40:07 dist-test-slave-zl47 krb5kdc[23282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:07.876180 23295 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:07.876914 23295 webserver.cc:460] Webserver started at http://127.0.0.1:37269/ using document root <none> and password file <none>
I20260923 14:40:07.877355 23295 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:07.877439 23295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:07.877627 23295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:07.879299 23295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "6f9d02cb5bc941b39ac5044a010c41dc"
format_stamp: "Formatted at 2026-09-23 14:40:07 on dist-test-slave-zl47"
server_key: "dde6701bf1a30a006dbe307c5b79fbdd"
server_key_iv: "97207b1c6b131ec49854386015e52770"
server_key_version: "encryptionkey@0"
I20260923 14:40:07.879765 23295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "6f9d02cb5bc941b39ac5044a010c41dc"
format_stamp: "Formatted at 2026-09-23 14:40:07 on dist-test-slave-zl47"
server_key: "dde6701bf1a30a006dbe307c5b79fbdd"
server_key_iv: "97207b1c6b131ec49854386015e52770"
server_key_version: "encryptionkey@0"
I20260923 14:40:07.882777 23295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:40:07.884656 23310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:07.885675 23295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:07.885792 23295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "6f9d02cb5bc941b39ac5044a010c41dc"
format_stamp: "Formatted at 2026-09-23 14:40:07 on dist-test-slave-zl47"
server_key: "dde6701bf1a30a006dbe307c5b79fbdd"
server_key_iv: "97207b1c6b131ec49854386015e52770"
server_key_version: "encryptionkey@0"
I20260923 14:40:07.885886 23295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:07.907687 23295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:07.909833 23295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:07.910000 23295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:07.916290 23295 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41011
I20260923 14:40:07.916364 23362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41011 every 8 connection(s)
I20260923 14:40:07.917059 23295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:07.919245 23363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:07.923782 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc: Bootstrap starting.
I20260923 14:40:07.925704 23363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:07.926282 23363 log.cc:824] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:07.926611 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 23295
I20260923 14:40:07.926769 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:07.927086 2920 external_mini_cluster.cc:1428] Setting key f7cc5a31db89202a47941a567153d1f7
I20260923 14:40:07.928295 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc: No bootstrap required, opened a new log
I20260923 14:40:07.931650 23363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9d02cb5bc941b39ac5044a010c41dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41011 } }
I20260923 14:40:07.931908 23363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:07.931967 23363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9d02cb5bc941b39ac5044a010c41dc, State: Initialized, Role: FOLLOWER
I20260923 14:40:07.932305 23363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9d02cb5bc941b39ac5044a010c41dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41011 } }
I20260923 14:40:07.932439 23363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:07.932523 23363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:07.932642 23363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:07.933506 23363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9d02cb5bc941b39ac5044a010c41dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41011 } }
I20260923 14:40:07.933790 23363 leader_election.cc:302] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f9d02cb5bc941b39ac5044a010c41dc; no voters:
I20260923 14:40:07.934084 23363 leader_election.cc:288] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:07.934309 23368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:07.934573 23368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [term 1 LEADER]: Becoming Leader. State: Replica: 6f9d02cb5bc941b39ac5044a010c41dc, State: Running, Role: LEADER
I20260923 14:40:07.934876 23368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9d02cb5bc941b39ac5044a010c41dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41011 } }
I20260923 14:40:07.935066 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:07.928126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34791 (local address 127.0.0.1:41011)
0923 14:40:07.928463 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:07.928486 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:07.928515 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:40:07.929056 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:07.929069 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:07.929326 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:07.929559 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:07.929875 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:07.930757 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:07.931842 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:07.932398 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:07.932583 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:07.933285 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:07.933290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:07.933300 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:07.933659 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:07.933715 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:07.933737 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:07.933833 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:07.934083 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 14:40:07.934223 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":111,"threads_started":1}
I20260923 14:40:07.935130 23363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:07.936429 23369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f9d02cb5bc941b39ac5044a010c41dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9d02cb5bc941b39ac5044a010c41dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41011 } } }
I20260923 14:40:07.936479 23370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f9d02cb5bc941b39ac5044a010c41dc. Latest consensus state: current_term: 1 leader_uuid: "6f9d02cb5bc941b39ac5044a010c41dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9d02cb5bc941b39ac5044a010c41dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41011 } } }
I20260923 14:40:07.936545 23369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:07.936573 23370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:07.937678 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 14:40:07.939330 23383 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:40:07.939401 23383 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:40:07.939613 23380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:07.940560 23380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:07.943686 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:07.939543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50889 (local address 127.0.0.1:41011)
0923 14:40:07.939642 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:07.939647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:07.939810 (+ 163us) server_negotiation.cc:407] Connection header received
0923 14:40:07.939915 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:07.939919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:07.939959 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:07.940039 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:07.940281 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:07.940881 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:07.941961 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:07.942300 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:07.942428 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:07.943244 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:07.943247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:07.943250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:07.943261 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:07.943292 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:07.943303 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:07.943401 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:07.943519 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 14:40:07.943548 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:40:07.945636 23380 catalog_manager.cc:1347] Generated new cluster ID: 8a609efd66594b98a41893a4d45c1576
I20260923 14:40:07.945703 23380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:07.953024 23380 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:07.953749 23380 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:07.959329 23380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f9d02cb5bc941b39ac5044a010c41dc: Generated new TSK 0
I20260923 14:40:07.959834 23380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:07.965490 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 23295
W20260923 14:40:07.970192 23280 connection.cc:535] client connection to 127.0.0.1:41011 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41011): Connection reset by peer (error 104)
2026-09-23T14:40:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:08 dist-test-slave-zl47 krb5kdc[23394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:08 dist-test-slave-zl47 krb5kdc[23394](info): set up 2 sockets
Sep 23 14:40:08 dist-test-slave-zl47 krb5kdc[23394](info): commencing operation
krb5kdc: starting...
W20260923 14:40:08.862298 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.878s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:08 dist-test-slave-zl47 krb5kdc[23394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:08.965664 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:09.057212 23407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:09.057438 23407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:09.057502 23407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:09.060853 23407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:09.060925 23407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:09.060958 23407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:09.061008 23407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:09.061043 23407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:09.065546 23407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:09.066466 23407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:09.067291 23407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:09.074312 23412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:09.074383 23407 server_base.cc:1034] running on GCE node
W20260923 14:40:09.074350 23415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:09.074311 23413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:09.075048 23407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:09.075897 23407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:09.077045 23407 hybrid_clock.cc:648] HybridClock initialized: now 1790174409077029 us; error 26 us; skew 500 ppm
Sep 23 14:40:09 dist-test-slave-zl47 krb5kdc[23394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:09.079401 23407 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:09.080112 23407 webserver.cc:460] Webserver started at http://127.0.0.1:36949/ using document root <none> and password file <none>
I20260923 14:40:09.080556 23407 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:09.080606 23407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:09.080806 23407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:09.082468 23407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "230ba604118d454cad763ca306b581b6"
format_stamp: "Formatted at 2026-09-23 14:40:09 on dist-test-slave-zl47"
server_key: "bc17b9570388458b9389fd751eea37a7"
server_key_iv: "4240582cf8a2c52ff1719fa7d894971f"
server_key_version: "encryptionkey@0"
I20260923 14:40:09.082995 23407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "230ba604118d454cad763ca306b581b6"
format_stamp: "Formatted at 2026-09-23 14:40:09 on dist-test-slave-zl47"
server_key: "bc17b9570388458b9389fd751eea37a7"
server_key_iv: "4240582cf8a2c52ff1719fa7d894971f"
server_key_version: "encryptionkey@0"
I20260923 14:40:09.086071 23407 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 14:40:09.088053 23422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:09.089026 23407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:09.089151 23407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "230ba604118d454cad763ca306b581b6"
format_stamp: "Formatted at 2026-09-23 14:40:09 on dist-test-slave-zl47"
server_key: "bc17b9570388458b9389fd751eea37a7"
server_key_iv: "4240582cf8a2c52ff1719fa7d894971f"
server_key_version: "encryptionkey@0"
I20260923 14:40:09.089243 23407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:09.092975 23407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:09.095530 23407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:09.095710 23407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:09.102324 23407 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36955
I20260923 14:40:09.102346 23474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36955 every 8 connection(s)
I20260923 14:40:09.103022 23407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:09.105362 23475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:09.109892 23475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6: Bootstrap starting.
I20260923 14:40:09.109977 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 23407
I20260923 14:40:09.110147 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:09.110394 2920 external_mini_cluster.cc:1428] Setting key 963d937d29a26fa1b9a3d75f34c01d8d
I20260923 14:40:09.112033 23475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:09.112645 23475 log.cc:824] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:09.114462 23475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6: No bootstrap required, opened a new log
I20260923 14:40:09.114773 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:09.111484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41565 (local address 127.0.0.1:36955)
0923 14:40:09.111876 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:09.111888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:09.111917 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:40:09.112549 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:09.112564 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:09.112833 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:09.113139 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:09.113714 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:09.113718 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:09.113727 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:09.113734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:09.113766 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:09.113780 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:09.113849 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:09.113979 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 14:40:09.114141 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":99,"threads_started":1}
I20260923 14:40:09.117074 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:09.117940 23475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230ba604118d454cad763ca306b581b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260923 14:40:09.118155 23475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:09.118235 23475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 230ba604118d454cad763ca306b581b6, State: Initialized, Role: FOLLOWER
I20260923 14:40:09.118599 23475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230ba604118d454cad763ca306b581b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260923 14:40:09.118701 23475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:09.118772 23475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:09.118860 23475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:09.119845 23475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230ba604118d454cad763ca306b581b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260923 14:40:09.120175 23475 leader_election.cc:302] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 230ba604118d454cad763ca306b581b6; no voters:
I20260923 14:40:09.120363 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:09.118856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49959 (local address 127.0.0.1:36955)
0923 14:40:09.118994 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:09.118998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:09.119185 (+ 187us) server_negotiation.cc:407] Connection header received
0923 14:40:09.119271 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:09.119273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:09.119294 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:09.119346 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:09.119953 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:09.119985 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:09.119992 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:09.120003 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:09.120027 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:09.120037 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:09.120111 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:09.120219 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 14:40:09.120244 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 14:40:09.120486 23475 leader_election.cc:288] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:09.120558 23485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:09.120731 23485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [term 1 LEADER]: Becoming Leader. State: Replica: 230ba604118d454cad763ca306b581b6, State: Running, Role: LEADER
I20260923 14:40:09.121017 23485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230ba604118d454cad763ca306b581b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } }
I20260923 14:40:09.121512 23475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:09.122442 23486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "230ba604118d454cad763ca306b581b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230ba604118d454cad763ca306b581b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } } }
I20260923 14:40:09.122546 23486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:09.122862 23494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:09.122851 23487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 230ba604118d454cad763ca306b581b6. Latest consensus state: current_term: 1 leader_uuid: "230ba604118d454cad763ca306b581b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "230ba604118d454cad763ca306b581b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36955 } } }
I20260923 14:40:09.123497 23487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:09.125552 23494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:09.130079 23494 catalog_manager.cc:1347] Generated new cluster ID: b82d3363e4534021b27b4d5eb31aa6ca
I20260923 14:40:09.130208 23494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:09.140434 23494 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:09.140671 23494 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:09.145130 23494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 230ba604118d454cad763ca306b581b6: Generated new TSK 0
I20260923 14:40:09.145699 23494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:09.149540 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 23407
2026-09-23T14:40:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1183 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:09 dist-test-slave-zl47 krb5kdc[23506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:09 dist-test-slave-zl47 krb5kdc[23506](info): set up 2 sockets
Sep 23 14:40:09 dist-test-slave-zl47 krb5kdc[23506](info): commencing operation
krb5kdc: starting...
W20260923 14:40:10.058266 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.890s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:10 dist-test-slave-zl47 krb5kdc[23506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:10.159093 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:10.253046 23519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:10.253237 23519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:10.253269 23519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:10.256690 23519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:10.256732 23519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:10.256748 23519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:10.256762 23519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:10.256774 23519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:10.261135 23519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:10.261922 23519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:10.262717 23519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:10.269806 23525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:10.269848 23524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:10.269848 23527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:10.270195 23519 server_base.cc:1034] running on GCE node
I20260923 14:40:10.270714 23519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:10.271564 23519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:10.272739 23519 hybrid_clock.cc:648] HybridClock initialized: now 1790174410272723 us; error 29 us; skew 500 ppm
Sep 23 14:40:10 dist-test-slave-zl47 krb5kdc[23506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:10.275113 23519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:10.275769 23519 webserver.cc:460] Webserver started at http://127.0.0.1:45069/ using document root <none> and password file <none>
I20260923 14:40:10.276211 23519 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:10.276284 23519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:10.276463 23519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:10.278050 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "5bc18a4b995642b0b61dcf8f81a98dfc"
format_stamp: "Formatted at 2026-09-23 14:40:10 on dist-test-slave-zl47"
server_key: "6799d7cafa8acf18d526b29b3ed077d5"
server_key_iv: "73d9e352dc43694cc60db2b1c5422bab"
server_key_version: "encryptionkey@0"
I20260923 14:40:10.278544 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "5bc18a4b995642b0b61dcf8f81a98dfc"
format_stamp: "Formatted at 2026-09-23 14:40:10 on dist-test-slave-zl47"
server_key: "6799d7cafa8acf18d526b29b3ed077d5"
server_key_iv: "73d9e352dc43694cc60db2b1c5422bab"
server_key_version: "encryptionkey@0"
I20260923 14:40:10.281416 23519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:40:10.283360 23534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:10.284394 23519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:40:10.284534 23519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "5bc18a4b995642b0b61dcf8f81a98dfc"
format_stamp: "Formatted at 2026-09-23 14:40:10 on dist-test-slave-zl47"
server_key: "6799d7cafa8acf18d526b29b3ed077d5"
server_key_iv: "73d9e352dc43694cc60db2b1c5422bab"
server_key_version: "encryptionkey@0"
I20260923 14:40:10.284636 23519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:10.298590 23519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:10.301365 23519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:10.301561 23519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:10.307802 23519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41355
I20260923 14:40:10.307813 23586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41355 every 8 connection(s)
I20260923 14:40:10.308562 23519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:10.310773 23587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:10.313565 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 23519
I20260923 14:40:10.313694 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:10.313956 2920 external_mini_cluster.cc:1428] Setting key 4db3fde0d0a0e532ff0c98b114fa5dff
I20260923 14:40:10.315636 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc: Bootstrap starting.
I20260923 14:40:10.317667 23587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:10.318413 23587 log.cc:824] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:10.320132 23587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc: No bootstrap required, opened a new log
Sep 23 14:40:10 dist-test-slave-zl47 krb5kdc[23506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 14:40:10.323379 23587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc18a4b995642b0b61dcf8f81a98dfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260923 14:40:10.323571 23587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:10.323639 23587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc18a4b995642b0b61dcf8f81a98dfc, State: Initialized, Role: FOLLOWER
I20260923 14:40:10.324071 23587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc18a4b995642b0b61dcf8f81a98dfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260923 14:40:10.324218 23587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:10.324288 23587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:10.324385 23587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:10.325254 23587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc18a4b995642b0b61dcf8f81a98dfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260923 14:40:10.325557 23587 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bc18a4b995642b0b61dcf8f81a98dfc; no voters:
I20260923 14:40:10.325853 23587 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:10.325912 23592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:10.326079 23592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [term 1 LEADER]: Becoming Leader. State: Replica: 5bc18a4b995642b0b61dcf8f81a98dfc, State: Running, Role: LEADER
I20260923 14:40:10.326059 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:10.315108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37007 (local address 127.0.0.1:41355)
0923 14:40:10.315448 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:10.315462 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:10.315493 (+ 31us) server_negotiation.cc:407] Connection header received
0923 14:40:10.316083 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:10.316098 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:10.316373 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:10.316608 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:10.317332 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:10.318379 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:10.319262 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:10.319653 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:10.319764 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:10.322174 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:10.322178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:10.322187 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:10.322208 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:10.323603 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:10.324019 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:10.324023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:10.324026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:10.324117 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:10.324381 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:10.324383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:10.324385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:10.324714 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:10.324838 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:10.325076 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 14:40:10.325295 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":108,"threads_started":1}
I20260923 14:40:10.326402 23592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc18a4b995642b0b61dcf8f81a98dfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } }
I20260923 14:40:10.326922 23587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:10.327509 23592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc18a4b995642b0b61dcf8f81a98dfc. Latest consensus state: current_term: 1 leader_uuid: "5bc18a4b995642b0b61dcf8f81a98dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc18a4b995642b0b61dcf8f81a98dfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } } }
I20260923 14:40:10.327684 23592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:10.327834 23593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bc18a4b995642b0b61dcf8f81a98dfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc18a4b995642b0b61dcf8f81a98dfc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41355 } } }
I20260923 14:40:10.327941 23593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:10.328588 23600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:10.329102 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:10.331209 23600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:10.335829 23600 catalog_manager.cc:1347] Generated new cluster ID: 1bdf2ff3d938412488662efcf5affb92
I20260923 14:40:10.335925 23600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:10.339345 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:10.331012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53565 (local address 127.0.0.1:41355)
0923 14:40:10.331093 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:10.331098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:10.332650 (+ 1552us) server_negotiation.cc:407] Connection header received
0923 14:40:10.332745 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:10.332749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:10.332788 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:10.332861 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:10.333394 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:10.333828 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:10.334772 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:10.334978 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:10.335070 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:10.336499 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:10.336522 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:10.336524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:10.336547 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:10.337926 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:10.338337 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:10.338340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:10.338342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:10.338416 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:10.338651 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:10.338654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:10.338655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:10.338828 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:10.338948 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:10.339128 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 14:40:10.339222 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 14:40:10.346334 23600 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:10.347170 23600 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:10.356213 23600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bc18a4b995642b0b61dcf8f81a98dfc: Generated new TSK 0
I20260923 14:40:10.356750 23600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:10.365478 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 23519
2026-09-23T14:40:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:10 dist-test-slave-zl47 krb5kdc[23618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:10 dist-test-slave-zl47 krb5kdc[23618](info): set up 2 sockets
Sep 23 14:40:10 dist-test-slave-zl47 krb5kdc[23618](info): commencing operation
krb5kdc: starting...
W20260923 14:40:11.274313 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.890s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:11 dist-test-slave-zl47 krb5kdc[23618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:11.377117 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37869
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:11.470830 23631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:11.471024 23631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:11.471058 23631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:11.474383 23631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:11.474423 23631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:11.474457 23631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:11.474471 23631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:11.474529 23631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:11.478797 23631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:11.479575 23631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:11.480396 23631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:11.487883 23637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:11.488041 23631 server_base.cc:1034] running on GCE node
W20260923 14:40:11.487895 23636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:11.487891 23639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:11.488662 23631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:11.489655 23631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:11.490826 23631 hybrid_clock.cc:648] HybridClock initialized: now 1790174411490799 us; error 39 us; skew 500 ppm
Sep 23 14:40:11 dist-test-slave-zl47 krb5kdc[23618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:11.498342 23631 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:11.499060 23631 webserver.cc:460] Webserver started at http://127.0.0.1:41427/ using document root <none> and password file <none>
I20260923 14:40:11.499531 23631 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:11.499607 23631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:11.499799 23631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:11.501529 23631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "dfbf5b96414a4c3cb91730a634128c48"
format_stamp: "Formatted at 2026-09-23 14:40:11 on dist-test-slave-zl47"
server_key: "21a68383f2cc0f098c7b5ca0aad30cfb"
server_key_iv: "17bbf8e211e2edddaf0bbb20ee07b639"
server_key_version: "encryptionkey@0"
I20260923 14:40:11.502000 23631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "dfbf5b96414a4c3cb91730a634128c48"
format_stamp: "Formatted at 2026-09-23 14:40:11 on dist-test-slave-zl47"
server_key: "21a68383f2cc0f098c7b5ca0aad30cfb"
server_key_iv: "17bbf8e211e2edddaf0bbb20ee07b639"
server_key_version: "encryptionkey@0"
I20260923 14:40:11.505043 23631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260923 14:40:11.507117 23646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:11.508106 23631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:40:11.508226 23631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "dfbf5b96414a4c3cb91730a634128c48"
format_stamp: "Formatted at 2026-09-23 14:40:11 on dist-test-slave-zl47"
server_key: "21a68383f2cc0f098c7b5ca0aad30cfb"
server_key_iv: "17bbf8e211e2edddaf0bbb20ee07b639"
server_key_version: "encryptionkey@0"
I20260923 14:40:11.508337 23631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:11.526252 23631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:11.528558 23631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:11.528724 23631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:11.534628 23631 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37869
I20260923 14:40:11.534646 23698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37869 every 8 connection(s)
I20260923 14:40:11.535344 23631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:11.537658 23699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:11.541567 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 23631
I20260923 14:40:11.541708 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:11.541949 2920 external_mini_cluster.cc:1428] Setting key 0b8ca9a9d8e62523a651768a80f926d1
I20260923 14:40:11.542421 23699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48: Bootstrap starting.
I20260923 14:40:11.544798 23699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:11.545540 23699 log.cc:824] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:11.547266 23699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48: No bootstrap required, opened a new log
I20260923 14:40:11.550194 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:11.543098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57133 (local address 127.0.0.1:37869)
0923 14:40:11.543469 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:11.543490 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:11.543527 (+ 37us) server_negotiation.cc:407] Connection header received
0923 14:40:11.544227 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:11.544247 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:11.544613 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:11.544931 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:11.545290 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:11.546465 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:11.547233 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:11.547820 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:11.547958 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:11.548612 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:11.548618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:11.548631 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:11.549037 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:11.549094 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:11.549118 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:11.549239 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:11.549408 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 14:40:11.549583 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":90,"threads_started":1}
I20260923 14:40:11.551071 23699 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf5b96414a4c3cb91730a634128c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } }
I20260923 14:40:11.551293 23699 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:11.551342 23699 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbf5b96414a4c3cb91730a634128c48, State: Initialized, Role: FOLLOWER
I20260923 14:40:11.551749 23699 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf5b96414a4c3cb91730a634128c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } }
I20260923 14:40:11.551914 23699 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:11.552007 23699 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:11.552100 23699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:11.552703 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:11.553066 23699 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf5b96414a4c3cb91730a634128c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } }
I20260923 14:40:11.553359 23699 leader_election.cc:302] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfbf5b96414a4c3cb91730a634128c48; no voters:
I20260923 14:40:11.553634 23699 leader_election.cc:288] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:11.553706 23704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:11.553910 23704 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [term 1 LEADER]: Becoming Leader. State: Replica: dfbf5b96414a4c3cb91730a634128c48, State: Running, Role: LEADER
I20260923 14:40:11.554219 23704 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf5b96414a4c3cb91730a634128c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } }
I20260923 14:40:11.554744 23699 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:11.555656 23709 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfbf5b96414a4c3cb91730a634128c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf5b96414a4c3cb91730a634128c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } } }
I20260923 14:40:11.555985 23709 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:11.555893 23708 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfbf5b96414a4c3cb91730a634128c48. Latest consensus state: current_term: 1 leader_uuid: "dfbf5b96414a4c3cb91730a634128c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbf5b96414a4c3cb91730a634128c48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37869 } } }
I20260923 14:40:11.556272 23708 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:11.558933 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:11.554743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49669 (local address 127.0.0.1:37869)
0923 14:40:11.554832 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:11.554836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:11.554845 (+ 9us) server_negotiation.cc:407] Connection header received
0923 14:40:11.554871 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:11.554873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:11.554915 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:11.554992 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:11.555582 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:11.556285 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:11.557524 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:11.557756 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:11.557865 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:11.558562 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:11.558565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:11.558567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:11.558577 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:11.558599 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:11.558610 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:11.558680 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:11.558756 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 14:40:11.558794 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260923 14:40:11.559248 23724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:40:11.559315 23724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:40:11.559397 23718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:11.560098 23718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:11.564419 23718 catalog_manager.cc:1347] Generated new cluster ID: 5181b402a72a44fd8e9e196da7e8d2bd
I20260923 14:40:11.564472 23718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:11.578487 23718 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:11.579177 23718 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:11.583039 23718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfbf5b96414a4c3cb91730a634128c48: Generated new TSK 0
I20260923 14:40:11.583645 23718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:11.592118 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 23631
2026-09-23T14:40:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:11 dist-test-slave-zl47 krb5kdc[23730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:11 dist-test-slave-zl47 krb5kdc[23730](info): set up 2 sockets
Sep 23 14:40:11 dist-test-slave-zl47 krb5kdc[23730](info): commencing operation
krb5kdc: starting...
W20260923 14:40:12.510324 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.900s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:12 dist-test-slave-zl47 krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:12.611033 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:12.703306 23743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:12.703498 23743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:12.703532 23743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:12.707039 23743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:12.707082 23743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:12.707099 23743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:12.707114 23743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:12.707127 23743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:12.711477 23743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:12.712260 23743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:12.713027 23743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:12.720248 23748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:12.720372 23751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:12.720311 23743 server_base.cc:1034] running on GCE node
W20260923 14:40:12.720257 23749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:12.720981 23743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:12.721930 23743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:12.723120 23743 hybrid_clock.cc:648] HybridClock initialized: now 1790174412723103 us; error 31 us; skew 500 ppm
Sep 23 14:40:12 dist-test-slave-zl47 krb5kdc[23730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:12.725608 23743 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:12.726404 23743 webserver.cc:460] Webserver started at http://127.0.0.1:43139/ using document root <none> and password file <none>
I20260923 14:40:12.726864 23743 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:12.726922 23743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:12.727144 23743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:12.728807 23743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "06ec4d27c198447fba6fdc9facf63c3f"
format_stamp: "Formatted at 2026-09-23 14:40:12 on dist-test-slave-zl47"
server_key: "0b0d423ebb11b72a0ed6b29c9d071b6d"
server_key_iv: "55bd313f4f11e0851318cce6f42e5610"
server_key_version: "encryptionkey@0"
I20260923 14:40:12.729310 23743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "06ec4d27c198447fba6fdc9facf63c3f"
format_stamp: "Formatted at 2026-09-23 14:40:12 on dist-test-slave-zl47"
server_key: "0b0d423ebb11b72a0ed6b29c9d071b6d"
server_key_iv: "55bd313f4f11e0851318cce6f42e5610"
server_key_version: "encryptionkey@0"
I20260923 14:40:12.732486 23743 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:40:12.734477 23758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:12.735458 23743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:12.735575 23743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "06ec4d27c198447fba6fdc9facf63c3f"
format_stamp: "Formatted at 2026-09-23 14:40:12 on dist-test-slave-zl47"
server_key: "0b0d423ebb11b72a0ed6b29c9d071b6d"
server_key_iv: "55bd313f4f11e0851318cce6f42e5610"
server_key_version: "encryptionkey@0"
I20260923 14:40:12.735667 23743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:12.739560 23743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:12.742395 23743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:12.742576 23743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:12.748553 23743 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40921
I20260923 14:40:12.748602 23810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40921 every 8 connection(s)
I20260923 14:40:12.749356 23743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:12.751564 23811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:12.755285 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 23743
I20260923 14:40:12.755422 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:12.755687 2920 external_mini_cluster.cc:1428] Setting key 21276814913b9d0024fc98b6b72d3147
I20260923 14:40:12.756127 23811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f: Bootstrap starting.
I20260923 14:40:12.758033 23811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:12.758828 23811 log.cc:824] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:12.760308 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:12.756956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53877 (local address 127.0.0.1:40921)
0923 14:40:12.757329 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:12.757341 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:12.757377 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:40:12.757900 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:12.757914 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:12.758224 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:12.758483 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:12.759248 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:12.759254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:12.759266 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:12.759278 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:12.759333 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:12.759372 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:12.759481 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:12.759598 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 14:40:12.759738 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":118,"threads_started":1}
I20260923 14:40:12.760564 23811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f: No bootstrap required, opened a new log
I20260923 14:40:12.762804 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:12.763898 23811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ec4d27c198447fba6fdc9facf63c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } }
I20260923 14:40:12.764129 23811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:12.764209 23811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06ec4d27c198447fba6fdc9facf63c3f, State: Initialized, Role: FOLLOWER
I20260923 14:40:12.764590 23811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ec4d27c198447fba6fdc9facf63c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } }
I20260923 14:40:12.764731 23811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:12.764849 23811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:12.764961 23811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:12.766004 23811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ec4d27c198447fba6fdc9facf63c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } }
I20260923 14:40:12.766268 23814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:12.764742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49345 (local address 127.0.0.1:40921)
0923 14:40:12.764845 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:12.764849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:12.765027 (+ 178us) server_negotiation.cc:407] Connection header received
0923 14:40:12.765143 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:12.765144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:12.765169 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:12.765230 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:12.765834 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:12.765837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:12.765838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:12.765847 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:12.765867 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:12.765883 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:12.765948 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:12.766078 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 14:40:12.766164 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 14:40:12.766314 23811 leader_election.cc:302] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06ec4d27c198447fba6fdc9facf63c3f; no voters:
I20260923 14:40:12.766636 23811 leader_election.cc:288] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:12.766731 23821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:12.766938 23821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [term 1 LEADER]: Becoming Leader. State: Replica: 06ec4d27c198447fba6fdc9facf63c3f, State: Running, Role: LEADER
I20260923 14:40:12.767262 23821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ec4d27c198447fba6fdc9facf63c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } }
I20260923 14:40:12.767661 23811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:12.768806 23822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06ec4d27c198447fba6fdc9facf63c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ec4d27c198447fba6fdc9facf63c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } } }
I20260923 14:40:12.768921 23822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:12.769114 23822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06ec4d27c198447fba6fdc9facf63c3f. Latest consensus state: current_term: 1 leader_uuid: "06ec4d27c198447fba6fdc9facf63c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06ec4d27c198447fba6fdc9facf63c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } } }
I20260923 14:40:12.769181 23822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:12.769245 23832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:12.771500 23832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:12.775652 23832 catalog_manager.cc:1347] Generated new cluster ID: 71614bafc8af48948b99e94341edf369
I20260923 14:40:12.775704 23832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:12.780421 23832 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:12.780686 23832 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:12.786652 23832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06ec4d27c198447fba6fdc9facf63c3f: Generated new TSK 0
I20260923 14:40:12.787117 23832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:12.792572 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 23743
2026-09-23T14:40:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:12 dist-test-slave-zl47 krb5kdc[23842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:12 dist-test-slave-zl47 krb5kdc[23842](info): set up 2 sockets
Sep 23 14:40:12 dist-test-slave-zl47 krb5kdc[23842](info): commencing operation
krb5kdc: starting...
W20260923 14:40:13.690589 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.879s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:13 dist-test-slave-zl47 krb5kdc[23842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:13.792659 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:13.883553 23855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:13.883728 23855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:13.883797 23855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:13.887217 23855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:13.887257 23855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:13.887274 23855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:13.887312 23855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:13.887341 23855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:13.891449 23855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:13.892171 23855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:13.892920 23855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:13.899876 23860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:13.900049 23855 server_base.cc:1034] running on GCE node
W20260923 14:40:13.899887 23863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:13.899958 23861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:13.900624 23855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:13.901399 23855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:13.902562 23855 hybrid_clock.cc:648] HybridClock initialized: now 1790174413902545 us; error 33 us; skew 500 ppm
Sep 23 14:40:13 dist-test-slave-zl47 krb5kdc[23842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:13.904850 23855 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:13.905615 23855 webserver.cc:460] Webserver started at http://127.0.0.1:42183/ using document root <none> and password file <none>
I20260923 14:40:13.905997 23855 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:13.906044 23855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:13.906245 23855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:13.907773 23855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "8baa78b000c04ae6bcf1493d8fead549"
format_stamp: "Formatted at 2026-09-23 14:40:13 on dist-test-slave-zl47"
server_key: "e6fbb30e78d9c9f15a3ffbad2178a070"
server_key_iv: "5ffcfa53e40e8aad1b37c9d6fcd25845"
server_key_version: "encryptionkey@0"
I20260923 14:40:13.908165 23855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "8baa78b000c04ae6bcf1493d8fead549"
format_stamp: "Formatted at 2026-09-23 14:40:13 on dist-test-slave-zl47"
server_key: "e6fbb30e78d9c9f15a3ffbad2178a070"
server_key_iv: "5ffcfa53e40e8aad1b37c9d6fcd25845"
server_key_version: "encryptionkey@0"
I20260923 14:40:13.910975 23855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 14:40:13.912820 23870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:13.913779 23855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:13.913959 23855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "8baa78b000c04ae6bcf1493d8fead549"
format_stamp: "Formatted at 2026-09-23 14:40:13 on dist-test-slave-zl47"
server_key: "e6fbb30e78d9c9f15a3ffbad2178a070"
server_key_iv: "5ffcfa53e40e8aad1b37c9d6fcd25845"
server_key_version: "encryptionkey@0"
I20260923 14:40:13.914068 23855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:13.928076 23855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:13.930454 23855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:13.930667 23855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:13.937536 23855 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43371
I20260923 14:40:13.937565 23922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43371 every 8 connection(s)
I20260923 14:40:13.938585 23855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:13.940835 23923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:13.945242 23923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549: Bootstrap starting.
I20260923 14:40:13.946766 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 23855
I20260923 14:40:13.946880 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:13.947108 2920 external_mini_cluster.cc:1428] Setting key ccd1992452f3e3db7015d1870b528a5a
I20260923 14:40:13.947362 23923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:13.948143 23923 log.cc:824] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:13.949694 23923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549: No bootstrap required, opened a new log
I20260923 14:40:13.953153 23923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa78b000c04ae6bcf1493d8fead549" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } }
I20260923 14:40:13.953367 23923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:13.953437 23923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8baa78b000c04ae6bcf1493d8fead549, State: Initialized, Role: FOLLOWER
I20260923 14:40:13.953791 23923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa78b000c04ae6bcf1493d8fead549" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } }
I20260923 14:40:13.953893 23923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:13.953967 23923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:13.954063 23923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 14:40:13 dist-test-slave-zl47 krb5kdc[23842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 14:40:13.954725 23923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa78b000c04ae6bcf1493d8fead549" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } }
I20260923 14:40:13.955001 23923 leader_election.cc:302] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8baa78b000c04ae6bcf1493d8fead549; no voters:
I20260923 14:40:13.955284 23923 leader_election.cc:288] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:13.955377 23928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:13.955557 23928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [term 1 LEADER]: Becoming Leader. State: Replica: 8baa78b000c04ae6bcf1493d8fead549, State: Running, Role: LEADER
I20260923 14:40:13.955830 23928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa78b000c04ae6bcf1493d8fead549" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } }
I20260923 14:40:13.956380 23923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:13.957195 23930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8baa78b000c04ae6bcf1493d8fead549. Latest consensus state: current_term: 1 leader_uuid: "8baa78b000c04ae6bcf1493d8fead549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa78b000c04ae6bcf1493d8fead549" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } } }
I20260923 14:40:13.957304 23930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:13.957580 23929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8baa78b000c04ae6bcf1493d8fead549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8baa78b000c04ae6bcf1493d8fead549" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43371 } } }
I20260923 14:40:13.957647 23929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:13.958351 23937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:13.960206 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:13.948059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:52999 (local address 127.0.0.1:43371)
0923 14:40:13.948408 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:13.948426 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:13.948462 (+ 36us) server_negotiation.cc:407] Connection header received
0923 14:40:13.948953 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:13.948966 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:13.949215 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:13.949431 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:13.950233 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:13.951020 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:13.952172 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:13.952774 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:13.952908 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:13.955331 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:13.955336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:13.955349 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:13.955382 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:13.957183 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:13.957775 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:13.957784 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:13.957792 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:13.957923 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:13.958279 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:13.958282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:13.958285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:13.958725 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:13.958894 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:13.959215 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 14:40:13.959463 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":114,"threads_started":1}
I20260923 14:40:13.960956 23937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:13.962766 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:13.966455 23937 catalog_manager.cc:1347] Generated new cluster ID: 4096abe1998c41cd805643f6216a6a50
I20260923 14:40:13.966550 23937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:13.972169 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:13.964698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:57935 (local address 127.0.0.1:43371)
0923 14:40:13.964782 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:13.964787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:13.965018 (+ 231us) server_negotiation.cc:407] Connection header received
0923 14:40:13.965121 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:13.965123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:13.965152 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:13.965233 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:13.965766 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:13.966267 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:13.967225 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:13.967553 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:13.967658 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:13.968928 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:13.968931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:13.968934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:13.968957 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:13.970382 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:13.970831 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:13.970834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:13.970836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:13.970924 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:13.971222 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:13.971225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:13.971227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:13.971407 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:13.971543 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:13.971776 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 14:40:13.972030 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 14:40:13.981958 23937 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:13.983057 23937 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:13.989123 23937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8baa78b000c04ae6bcf1493d8fead549: Generated new TSK 0
I20260923 14:40:13.989674 23937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:13.997202 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 23855
W20260923 14:40:14.002089 23840 connection.cc:535] client connection to 127.0.0.1:43371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43371): Connection reset by peer (error 104)
2026-09-23T14:40:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:14 dist-test-slave-zl47 krb5kdc[23954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:14 dist-test-slave-zl47 krb5kdc[23954](info): set up 2 sockets
Sep 23 14:40:14 dist-test-slave-zl47 krb5kdc[23954](info): commencing operation
krb5kdc: starting...
W20260923 14:40:14.890646 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.875s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:14 dist-test-slave-zl47 krb5kdc[23954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:14.988543 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:15.080808 23967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:15.080996 23967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:15.081029 23967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:15.084715 23967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:15.084759 23967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:15.084775 23967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:15.084815 23967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:15.084841 23967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:15.088984 23967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:15.089704 23967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:15.090502 23967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:15.097546 23972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:15.097576 23973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:15.097687 23967 server_base.cc:1034] running on GCE node
W20260923 14:40:15.097545 23975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:15.098275 23967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:15.099169 23967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:15.100342 23967 hybrid_clock.cc:648] HybridClock initialized: now 1790174415100322 us; error 34 us; skew 500 ppm
Sep 23 14:40:15 dist-test-slave-zl47 krb5kdc[23954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:15.109656 23967 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:15.110432 23967 webserver.cc:460] Webserver started at http://127.0.0.1:34675/ using document root <none> and password file <none>
I20260923 14:40:15.110883 23967 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:15.110960 23967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:15.111150 23967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:15.112905 23967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "beff67602cfb4b98b9d857274ab65f8e"
format_stamp: "Formatted at 2026-09-23 14:40:15 on dist-test-slave-zl47"
server_key: "f663ee51f939c2818434a2a391ef7249"
server_key_iv: "d951bcdc3b0f19f740abc1a578d7d8b9"
server_key_version: "encryptionkey@0"
I20260923 14:40:15.113394 23967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "beff67602cfb4b98b9d857274ab65f8e"
format_stamp: "Formatted at 2026-09-23 14:40:15 on dist-test-slave-zl47"
server_key: "f663ee51f939c2818434a2a391ef7249"
server_key_iv: "d951bcdc3b0f19f740abc1a578d7d8b9"
server_key_version: "encryptionkey@0"
I20260923 14:40:15.116514 23967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260923 14:40:15.118530 23982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:15.119529 23967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:15.119656 23967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "beff67602cfb4b98b9d857274ab65f8e"
format_stamp: "Formatted at 2026-09-23 14:40:15 on dist-test-slave-zl47"
server_key: "f663ee51f939c2818434a2a391ef7249"
server_key_iv: "d951bcdc3b0f19f740abc1a578d7d8b9"
server_key_version: "encryptionkey@0"
I20260923 14:40:15.119750 23967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:15.143901 23967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:15.146917 23967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:15.147100 23967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:15.153100 23967 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36385
I20260923 14:40:15.153189 24034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36385 every 8 connection(s)
I20260923 14:40:15.153899 23967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:15.156292 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:15.160959 24035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e: Bootstrap starting.
I20260923 14:40:15.163106 24035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:15.163233 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 23967
I20260923 14:40:15.163398 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:15.163653 2920 external_mini_cluster.cc:1428] Setting key dc49c47bd313e8abae1e8889bbc55863
I20260923 14:40:15.163921 24035 log.cc:824] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:15.165762 24035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e: No bootstrap required, opened a new log
I20260923 14:40:15.170025 24035 raft_consensus.cc:348] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beff67602cfb4b98b9d857274ab65f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260923 14:40:15.170284 24035 raft_consensus.cc:374] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:15.170349 24035 raft_consensus.cc:729] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beff67602cfb4b98b9d857274ab65f8e, State: Initialized, Role: FOLLOWER
I20260923 14:40:15.170807 24035 consensus_queue.cc:260] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beff67602cfb4b98b9d857274ab65f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260923 14:40:15.170948 24035 raft_consensus.cc:388] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:15.171007 24035 raft_consensus.cc:482] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:15.171087 24035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:15.172104 24035 raft_consensus.cc:504] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beff67602cfb4b98b9d857274ab65f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260923 14:40:15.172258 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:15.164595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:33239 (local address 127.0.0.1:36385)
0923 14:40:15.164953 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:15.164969 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:15.165008 (+ 39us) server_negotiation.cc:407] Connection header received
0923 14:40:15.165634 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:15.165654 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:15.166009 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:15.166377 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:15.166753 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:15.168113 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:15.169398 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:15.169815 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:15.169926 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:15.170791 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:15.170795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:15.170805 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:15.171195 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:15.171254 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:15.171274 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:15.171389 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:15.171578 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 14:40:15.171664 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260923 14:40:15.172449 24035 leader_election.cc:302] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beff67602cfb4b98b9d857274ab65f8e; no voters:
I20260923 14:40:15.172731 24035 leader_election.cc:288] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:15.172828 24040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:15.173022 24040 raft_consensus.cc:686] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [term 1 LEADER]: Becoming Leader. State: Replica: beff67602cfb4b98b9d857274ab65f8e, State: Running, Role: LEADER
I20260923 14:40:15.173758 24035 sys_catalog.cc:564] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:15.173965 24040 consensus_queue.cc:237] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beff67602cfb4b98b9d857274ab65f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260923 14:40:15.175160 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:15.175724 24041 sys_catalog.cc:455] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beff67602cfb4b98b9d857274ab65f8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beff67602cfb4b98b9d857274ab65f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } } }
I20260923 14:40:15.175909 24042 sys_catalog.cc:455] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader beff67602cfb4b98b9d857274ab65f8e. Latest consensus state: current_term: 1 leader_uuid: "beff67602cfb4b98b9d857274ab65f8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beff67602cfb4b98b9d857274ab65f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } } }
I20260923 14:40:15.176193 24042 sys_catalog.cc:458] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:15.176189 24041 sys_catalog.cc:458] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:15.176618 24055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:15.179352 24055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:15.182862 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:15.177504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:42989 (local address 127.0.0.1:36385)
0923 14:40:15.177603 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:15.177606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:15.177909 (+ 303us) server_negotiation.cc:407] Connection header received
0923 14:40:15.178009 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:15.178011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:15.178066 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:15.178168 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:15.178433 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:15.179062 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:15.180011 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:15.180301 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:15.180405 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:15.182366 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:15.182369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:15.182372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:15.182382 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:15.182413 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:15.182425 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:15.182513 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:15.182727 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 14:40:15.182756 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 14:40:15.183890 24055 catalog_manager.cc:1347] Generated new cluster ID: 22368bda6a2d4c12911af1dbd21ccb7a
I20260923 14:40:15.183941 24055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:15.191757 24055 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:15.192639 24055 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:15.207794 24055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beff67602cfb4b98b9d857274ab65f8e: Generated new TSK 0
I20260923 14:40:15.208348 24055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:15.219094 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 23967
W20260923 14:40:15.224798 23952 connection.cc:535] client connection to 127.0.0.1:36385 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36385): Connection reset by peer (error 104)
2026-09-23T14:40:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:15 dist-test-slave-zl47 krb5kdc[24066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:15 dist-test-slave-zl47 krb5kdc[24066](info): set up 2 sockets
Sep 23 14:40:15 dist-test-slave-zl47 krb5kdc[24066](info): commencing operation
krb5kdc: starting...
W20260923 14:40:16.122278 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.883s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:16 dist-test-slave-zl47 krb5kdc[24066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:16.230531 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:16.320814 24079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:16.321033 24079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:16.321096 24079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:16.324430 24079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:16.324499 24079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:16.324532 24079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:16.324563 24079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:16.324590 24079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:16.328984 24079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:16.329771 24079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:16.330667 24079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:16.337611 24084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:16.337651 24085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:16.337728 24079 server_base.cc:1034] running on GCE node
W20260923 14:40:16.337651 24087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:16.338282 24079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:16.339257 24079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:16.340435 24079 hybrid_clock.cc:648] HybridClock initialized: now 1790174416340421 us; error 31 us; skew 500 ppm
Sep 23 14:40:16 dist-test-slave-zl47 krb5kdc[24066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:16.342844 24079 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:16.343555 24079 webserver.cc:460] Webserver started at http://127.0.0.1:39919/ using document root <none> and password file <none>
I20260923 14:40:16.344038 24079 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:16.344118 24079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:16.344318 24079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:16.346031 24079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "93e7c275345f4be1b43a32a1fbf354ce"
format_stamp: "Formatted at 2026-09-23 14:40:16 on dist-test-slave-zl47"
server_key: "6514a1f06dbe4eef8a3d03c1efa7089a"
server_key_iv: "e648359d77d2e6df17bf62b476385bbb"
server_key_version: "encryptionkey@0"
I20260923 14:40:16.346619 24079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "93e7c275345f4be1b43a32a1fbf354ce"
format_stamp: "Formatted at 2026-09-23 14:40:16 on dist-test-slave-zl47"
server_key: "6514a1f06dbe4eef8a3d03c1efa7089a"
server_key_iv: "e648359d77d2e6df17bf62b476385bbb"
server_key_version: "encryptionkey@0"
I20260923 14:40:16.349664 24079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:40:16.351769 24094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:16.352864 24079 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260923 14:40:16.353005 24079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "93e7c275345f4be1b43a32a1fbf354ce"
format_stamp: "Formatted at 2026-09-23 14:40:16 on dist-test-slave-zl47"
server_key: "6514a1f06dbe4eef8a3d03c1efa7089a"
server_key_iv: "e648359d77d2e6df17bf62b476385bbb"
server_key_version: "encryptionkey@0"
I20260923 14:40:16.353127 24079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:16.357018 24079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:16.359886 24079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:16.360064 24079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:16.366221 24079 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42849
I20260923 14:40:16.366276 24146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42849 every 8 connection(s)
I20260923 14:40:16.366968 24079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:16.369097 24147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:16.373529 24147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce: Bootstrap starting.
I20260923 14:40:16.374637 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 24079
I20260923 14:40:16.374763 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:16.375059 2920 external_mini_cluster.cc:1428] Setting key 4f3e8bda479464c5a01729ebc58d22b0
I20260923 14:40:16.375866 24147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:16.376593 24147 log.cc:824] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:16.378175 24147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce: No bootstrap required, opened a new log
I20260923 14:40:16.379150 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:16.376025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:44783 (local address 127.0.0.1:42849)
0923 14:40:16.376389 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:16.376401 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:16.376429 (+ 28us) server_negotiation.cc:407] Connection header received
0923 14:40:16.376988 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:16.377002 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:16.377214 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:16.377492 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:16.378084 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:16.378088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:16.378124 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:16.378134 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:16.378189 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:16.378220 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:16.378308 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:16.378463 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 14:40:16.378592 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260923 14:40:16.381516 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:16.381419 24147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e7c275345f4be1b43a32a1fbf354ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } }
I20260923 14:40:16.381633 24147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:16.381677 24147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e7c275345f4be1b43a32a1fbf354ce, State: Initialized, Role: FOLLOWER
I20260923 14:40:16.382041 24147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e7c275345f4be1b43a32a1fbf354ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } }
I20260923 14:40:16.382192 24147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:16.382256 24147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:16.382344 24147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:16.383267 24147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e7c275345f4be1b43a32a1fbf354ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } }
I20260923 14:40:16.383591 24147 leader_election.cc:302] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93e7c275345f4be1b43a32a1fbf354ce; no voters:
I20260923 14:40:16.383922 24147 leader_election.cc:288] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:16.383942 24157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:16.384135 24157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [term 1 LEADER]: Becoming Leader. State: Replica: 93e7c275345f4be1b43a32a1fbf354ce, State: Running, Role: LEADER
I20260923 14:40:16.384431 24157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e7c275345f4be1b43a32a1fbf354ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } }
I20260923 14:40:16.384927 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:16.383436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:48577 (local address 127.0.0.1:42849)
0923 14:40:16.383543 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:16.383547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:16.383705 (+ 158us) server_negotiation.cc:407] Connection header received
0923 14:40:16.383796 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:16.383799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:16.383822 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:16.383920 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:16.384515 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:16.384518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:16.384520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:16.384532 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:16.384555 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:16.384567 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:16.384629 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:16.384753 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 14:40:16.384799 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:40:16.384958 24147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:16.386010 24158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93e7c275345f4be1b43a32a1fbf354ce. Latest consensus state: current_term: 1 leader_uuid: "93e7c275345f4be1b43a32a1fbf354ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e7c275345f4be1b43a32a1fbf354ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } } }
I20260923 14:40:16.386006 24159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93e7c275345f4be1b43a32a1fbf354ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93e7c275345f4be1b43a32a1fbf354ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42849 } } }
I20260923 14:40:16.386145 24159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:16.386145 24158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:16.386534 24168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:16.388757 24168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:16.392704 24168 catalog_manager.cc:1347] Generated new cluster ID: 8014e62d279d400cbc6d0e370a859b6f
I20260923 14:40:16.392756 24168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:16.412226 24168 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:16.412460 24168 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:16.416812 24168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93e7c275345f4be1b43a32a1fbf354ce: Generated new TSK 0
I20260923 14:40:16.417392 24168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:16.425526 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 24079
2026-09-23T14:40:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:16 dist-test-slave-zl47 krb5kdc[24178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:16 dist-test-slave-zl47 krb5kdc[24178](info): set up 2 sockets
Sep 23 14:40:16 dist-test-slave-zl47 krb5kdc[24178](info): commencing operation
krb5kdc: starting...
W20260923 14:40:17.318482 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.875s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:17 dist-test-slave-zl47 krb5kdc[24178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:17.418915 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:17.510031 24191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:17.510275 24191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:17.510345 24191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:17.513964 24191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:17.514060 24191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:17.514092 24191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:17.514155 24191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:17.514189 24191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:17.518384 24191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:17.519186 24191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:17.520021 24191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:17.527098 24197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:17.527093 24199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:17.527092 24196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:17.527428 24191 server_base.cc:1034] running on GCE node
I20260923 14:40:17.527880 24191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:17.528714 24191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:17.529887 24191 hybrid_clock.cc:648] HybridClock initialized: now 1790174417529877 us; error 29 us; skew 500 ppm
Sep 23 14:40:17 dist-test-slave-zl47 krb5kdc[24178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:17.536521 24191 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:17.537248 24191 webserver.cc:460] Webserver started at http://127.0.0.1:42101/ using document root <none> and password file <none>
I20260923 14:40:17.537695 24191 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:17.537771 24191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:17.537961 24191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:17.539601 24191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "df25fa56bee54cf088c8825783ad568a"
format_stamp: "Formatted at 2026-09-23 14:40:17 on dist-test-slave-zl47"
server_key: "680b2673a5afaf489f93f89da9eb2835"
server_key_iv: "9a6fc03b9f2a74d97aee3e29ad367e9f"
server_key_version: "encryptionkey@0"
I20260923 14:40:17.540067 24191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "df25fa56bee54cf088c8825783ad568a"
format_stamp: "Formatted at 2026-09-23 14:40:17 on dist-test-slave-zl47"
server_key: "680b2673a5afaf489f93f89da9eb2835"
server_key_iv: "9a6fc03b9f2a74d97aee3e29ad367e9f"
server_key_version: "encryptionkey@0"
I20260923 14:40:17.543067 24191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:40:17.544941 24206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:17.545902 24191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:17.546020 24191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "df25fa56bee54cf088c8825783ad568a"
format_stamp: "Formatted at 2026-09-23 14:40:17 on dist-test-slave-zl47"
server_key: "680b2673a5afaf489f93f89da9eb2835"
server_key_iv: "9a6fc03b9f2a74d97aee3e29ad367e9f"
server_key_version: "encryptionkey@0"
I20260923 14:40:17.546131 24191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:17.560006 24191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:17.562132 24191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:17.562291 24191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:17.568250 24191 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43739
I20260923 14:40:17.568265 24258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43739 every 8 connection(s)
I20260923 14:40:17.568948 24191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:17.571112 24259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:17.573532 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 24191
I20260923 14:40:17.573717 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:17.574005 2920 external_mini_cluster.cc:1428] Setting key 42210c598f858562b5b9d2b783c1021f
I20260923 14:40:17.575744 24259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a: Bootstrap starting.
I20260923 14:40:17.577684 24259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:17.578348 24259 log.cc:824] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:17.579973 24259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a: No bootstrap required, opened a new log
Sep 23 14:40:17 dist-test-slave-zl47 krb5kdc[24178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 14:40:17.583232 24259 raft_consensus.cc:348] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df25fa56bee54cf088c8825783ad568a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } }
I20260923 14:40:17.583419 24259 raft_consensus.cc:374] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:17.583496 24259 raft_consensus.cc:729] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df25fa56bee54cf088c8825783ad568a, State: Initialized, Role: FOLLOWER
I20260923 14:40:17.583871 24259 consensus_queue.cc:260] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df25fa56bee54cf088c8825783ad568a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } }
I20260923 14:40:17.584000 24259 raft_consensus.cc:388] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:17.584074 24259 raft_consensus.cc:482] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:17.584156 24259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:17.584792 24259 raft_consensus.cc:504] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df25fa56bee54cf088c8825783ad568a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } }
I20260923 14:40:17.585068 24259 leader_election.cc:302] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df25fa56bee54cf088c8825783ad568a; no voters:
I20260923 14:40:17.585328 24259 leader_election.cc:288] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:17.585381 24264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:17.585582 24264 raft_consensus.cc:686] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [term 1 LEADER]: Becoming Leader. State: Replica: df25fa56bee54cf088c8825783ad568a, State: Running, Role: LEADER
I20260923 14:40:17.585862 24264 consensus_queue.cc:237] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df25fa56bee54cf088c8825783ad568a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } }
I20260923 14:40:17.586367 24259 sys_catalog.cc:564] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:17.587329 24265 sys_catalog.cc:455] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df25fa56bee54cf088c8825783ad568a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df25fa56bee54cf088c8825783ad568a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } } }
I20260923 14:40:17.587358 24266 sys_catalog.cc:455] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df25fa56bee54cf088c8825783ad568a. Latest consensus state: current_term: 1 leader_uuid: "df25fa56bee54cf088c8825783ad568a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df25fa56bee54cf088c8825783ad568a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43739 } } }
I20260923 14:40:17.587527 24266 sys_catalog.cc:458] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:17.587488 24265 sys_catalog.cc:458] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:17.587550 24262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:17.575256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:46379 (local address 127.0.0.1:43739)
0923 14:40:17.575611 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:17.575628 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:17.575657 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:40:17.576216 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:17.576231 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:17.576518 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:17.576744 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:17.577509 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:17.578472 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:17.579450 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:17.579826 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:17.579932 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:17.582383 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:17.582388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:17.582397 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:17.582420 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:17.584094 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:17.584583 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:17.584588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:17.584593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:17.584701 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:17.585086 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:17.585089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:17.585090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:17.585633 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:17.586335 (+ 702us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:17.586538 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 14:40:17.586683 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":123,"threads_started":1}
I20260923 14:40:17.590222 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260923 14:40:17.590682 24279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:40:17.590739 24279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:40:17.590797 24274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:17.591569 24274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:17.596252 24274 catalog_manager.cc:1347] Generated new cluster ID: 91c97d513f604ea9ae4af843e3a71969
I20260923 14:40:17.596306 24274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:17.600980 24262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:17.592123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:60375 (local address 127.0.0.1:43739)
0923 14:40:17.593047 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:17.593051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:17.593066 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:40:17.593097 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:17.593099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:17.593136 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:17.593216 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:17.593980 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:17.595013 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:17.596010 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:17.596323 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:17.596410 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:17.597890 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:17.597894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:17.597896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:17.597919 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:17.599331 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:17.599835 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:17.599838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:17.599839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:17.599931 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:17.600241 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:17.600243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:17.600245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:17.600403 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:17.600524 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:17.600765 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 14:40:17.600841 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":871}
I20260923 14:40:17.606750 24274 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:17.607637 24274 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:17.616076 24274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df25fa56bee54cf088c8825783ad568a: Generated new TSK 0
I20260923 14:40:17.616611 24274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:17.627259 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 24191
W20260923 14:40:17.631799 24176 connection.cc:535] client connection to 127.0.0.1:43739 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43739): Connection reset by peer (error 104)
2026-09-23T14:40:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:17 dist-test-slave-zl47 krb5kdc[24290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:17 dist-test-slave-zl47 krb5kdc[24290](info): set up 2 sockets
Sep 23 14:40:17 dist-test-slave-zl47 krb5kdc[24290](info): commencing operation
krb5kdc: starting...
W20260923 14:40:18.562323 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:18 dist-test-slave-zl47 krb5kdc[24290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:18.664640 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:18.764983 24303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:18.765210 24303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:18.765280 24303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:18.769248 24303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:18.769300 24303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:18.769323 24303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:18.769383 24303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:18.769414 24303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:18.774086 24303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:18.775005 24303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:18.775826 24303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:18.783293 24309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:18.783290 24308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:18.783290 24311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:18.783883 24303 server_base.cc:1034] running on GCE node
I20260923 14:40:18.784307 24303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:18.785200 24303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:18.786347 24303 hybrid_clock.cc:648] HybridClock initialized: now 1790174418786344 us; error 48 us; skew 500 ppm
Sep 23 14:40:18 dist-test-slave-zl47 krb5kdc[24290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:18.788611 24303 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:18.789278 24303 webserver.cc:460] Webserver started at http://127.0.0.1:40979/ using document root <none> and password file <none>
I20260923 14:40:18.789676 24303 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:18.789722 24303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:18.789849 24303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:18.791535 24303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "4a6487597523498e817bcebbd46fbd5b"
format_stamp: "Formatted at 2026-09-23 14:40:18 on dist-test-slave-zl47"
server_key: "8f7970dbd063709436e0344b35057bd0"
server_key_iv: "0ce5b7ffb512ba370b0457d22d30bdf3"
server_key_version: "encryptionkey@0"
I20260923 14:40:18.791982 24303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "4a6487597523498e817bcebbd46fbd5b"
format_stamp: "Formatted at 2026-09-23 14:40:18 on dist-test-slave-zl47"
server_key: "8f7970dbd063709436e0344b35057bd0"
server_key_iv: "0ce5b7ffb512ba370b0457d22d30bdf3"
server_key_version: "encryptionkey@0"
I20260923 14:40:18.795043 24303 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:40:18.796895 24318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:18.797854 24303 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:18.797973 24303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "4a6487597523498e817bcebbd46fbd5b"
format_stamp: "Formatted at 2026-09-23 14:40:18 on dist-test-slave-zl47"
server_key: "8f7970dbd063709436e0344b35057bd0"
server_key_iv: "0ce5b7ffb512ba370b0457d22d30bdf3"
server_key_version: "encryptionkey@0"
I20260923 14:40:18.798064 24303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:18.805617 24303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:18.807756 24303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:18.807915 24303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:18.813683 24303 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40713
I20260923 14:40:18.813742 24370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40713 every 8 connection(s)
I20260923 14:40:18.814481 24303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:18.816697 24371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:18.819226 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 24303
I20260923 14:40:18.819345 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:18.819566 2920 external_mini_cluster.cc:1428] Setting key a5535af1fa495abe1cca1e611f2f51fa
I20260923 14:40:18.821427 24371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b: Bootstrap starting.
I20260923 14:40:18.823508 24371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:18.824244 24371 log.cc:824] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:18.825716 24371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b: No bootstrap required, opened a new log
I20260923 14:40:18.826977 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:18.820572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:45353 (local address 127.0.0.1:40713)
0923 14:40:18.820957 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:18.820971 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:18.821002 (+ 31us) server_negotiation.cc:407] Connection header received
0923 14:40:18.821613 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:18.821628 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:18.821926 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:18.822208 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:18.822494 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:18.823305 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:18.824418 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:18.824784 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:18.824933 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:18.825647 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:18.825651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:18.825660 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:18.825983 (+ 323us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:18.826027 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:18.826046 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:18.826196 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:18.826312 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 14:40:18.826420 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":123,"threads_started":1}
I20260923 14:40:18.829355 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:18.829278 24371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6487597523498e817bcebbd46fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } }
I20260923 14:40:18.829499 24371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:18.829547 24371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6487597523498e817bcebbd46fbd5b, State: Initialized, Role: FOLLOWER
I20260923 14:40:18.829864 24371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6487597523498e817bcebbd46fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } }
I20260923 14:40:18.829993 24371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:18.830047 24371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:18.830143 24371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:18.830899 24371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6487597523498e817bcebbd46fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } }
I20260923 14:40:18.831218 24371 leader_election.cc:302] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a6487597523498e817bcebbd46fbd5b; no voters:
I20260923 14:40:18.831568 24371 leader_election.cc:288] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:18.831660 24381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:18.831847 24381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [term 1 LEADER]: Becoming Leader. State: Replica: 4a6487597523498e817bcebbd46fbd5b, State: Running, Role: LEADER
I20260923 14:40:18.832132 24381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6487597523498e817bcebbd46fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } }
I20260923 14:40:18.832618 24371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:18.833633 24382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a6487597523498e817bcebbd46fbd5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6487597523498e817bcebbd46fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } } }
I20260923 14:40:18.833631 24383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a6487597523498e817bcebbd46fbd5b. Latest consensus state: current_term: 1 leader_uuid: "4a6487597523498e817bcebbd46fbd5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a6487597523498e817bcebbd46fbd5b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } } }
I20260923 14:40:18.833770 24383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:18.833770 24382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:18.834782 24390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:18.835616 24374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:18.831211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:33149 (local address 127.0.0.1:40713)
0923 14:40:18.831317 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:18.831321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:18.831441 (+ 120us) server_negotiation.cc:407] Connection header received
0923 14:40:18.831511 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:18.831516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:18.831558 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:18.831627 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:18.831998 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:18.832594 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:18.833791 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:18.834222 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:18.834308 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:18.835205 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:18.835208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:18.835209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:18.835218 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:18.835237 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:18.835246 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:18.835340 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:18.835476 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 14:40:18.835503 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:40:18.838548 24390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:18.843472 24390 catalog_manager.cc:1347] Generated new cluster ID: 84200f04adc14842bbe0233ec6d8b3f9
I20260923 14:40:18.843520 24390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:18.869481 24390 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:18.870266 24390 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:18.878000 24390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a6487597523498e817bcebbd46fbd5b: Generated new TSK 0
I20260923 14:40:18.878608 24390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:18.889576 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 24303
W20260923 14:40:18.894870 24288 connection.cc:535] client connection to 127.0.0.1:40713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40713): Connection reset by peer (error 104)
2026-09-23T14:40:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1263 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:18 dist-test-slave-zl47 krb5kdc[24402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:18 dist-test-slave-zl47 krb5kdc[24402](info): set up 2 sockets
Sep 23 14:40:18 dist-test-slave-zl47 krb5kdc[24402](info): commencing operation
krb5kdc: starting...
W20260923 14:40:19.866638 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:19 dist-test-slave-zl47 krb5kdc[24402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 14:40:19.976909 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:20.080262 24415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:20.080476 24415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:20.080513 24415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:20.084031 24415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:20.084082 24415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:20.084103 24415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:20.084120 24415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:20.084136 24415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:20.088780 24415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:20.089548 24415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:20.090369 24415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:20.097601 24423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:20.097642 24420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:20.097698 24421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:20.097736 24415 server_base.cc:1034] running on GCE node
I20260923 14:40:20.098322 24415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:20.099308 24415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:20.100468 24415 hybrid_clock.cc:648] HybridClock initialized: now 1790174420100450 us; error 31 us; skew 500 ppm
Sep 23 14:40:20 dist-test-slave-zl47 krb5kdc[24402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:20.103397 24415 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 14:40:20.104377 24415 webserver.cc:460] Webserver started at http://127.0.0.1:35423/ using document root <none> and password file <none>
I20260923 14:40:20.105014 24415 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:20.105131 24415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:20.105337 24415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:20.107468 24415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "ee0ced8835f14ccfb75d9ec5705fd365"
format_stamp: "Formatted at 2026-09-23 14:40:20 on dist-test-slave-zl47"
server_key: "fd7cc06e34bd4ac5eee2c41ec26fb494"
server_key_iv: "d1a11b597204c4348507ea5cceff2afc"
server_key_version: "encryptionkey@0"
I20260923 14:40:20.107983 24415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "ee0ced8835f14ccfb75d9ec5705fd365"
format_stamp: "Formatted at 2026-09-23 14:40:20 on dist-test-slave-zl47"
server_key: "fd7cc06e34bd4ac5eee2c41ec26fb494"
server_key_iv: "d1a11b597204c4348507ea5cceff2afc"
server_key_version: "encryptionkey@0"
I20260923 14:40:20.111181 24415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:40:20.113415 24430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:20.114535 24415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:40:20.114652 24415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "ee0ced8835f14ccfb75d9ec5705fd365"
format_stamp: "Formatted at 2026-09-23 14:40:20 on dist-test-slave-zl47"
server_key: "fd7cc06e34bd4ac5eee2c41ec26fb494"
server_key_iv: "d1a11b597204c4348507ea5cceff2afc"
server_key_version: "encryptionkey@0"
I20260923 14:40:20.114742 24415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:20.118729 24415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:20.121461 24415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:20.121620 24415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:20.128264 24415 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46335
I20260923 14:40:20.128393 24482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46335 every 8 connection(s)
I20260923 14:40:20.129216 24415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:20.131533 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 24415
I20260923 14:40:20.131654 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:20.131991 2920 external_mini_cluster.cc:1428] Setting key d756ea441e9760efc4c8ee34e8459ebe
I20260923 14:40:20.133623 24483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:20.139477 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:20.133369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:54149 (local address 127.0.0.1:46335)
0923 14:40:20.133940 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:20.133960 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:20.134385 (+ 425us) server_negotiation.cc:407] Connection header received
0923 14:40:20.135362 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:20.135405 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:20.135774 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:20.136133 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:20.137662 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:20.137670 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:20.137686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:20.137702 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:20.137775 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:20.137807 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:20.137966 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:20.138409 (+ 443us) server_negotiation.cc:299] Negotiation successful
0923 14:40:20.138571 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":133,"threads_started":1}
I20260923 14:40:20.141284 24483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365: Bootstrap starting.
I20260923 14:40:20.142336 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:20.145550 24486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:20.144172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:52159 (local address 127.0.0.1:46335)
0923 14:40:20.144295 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:20.144300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:20.144473 (+ 173us) server_negotiation.cc:407] Connection header received
0923 14:40:20.144555 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:20.144557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:20.144585 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:20.144662 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:20.145219 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:20.145223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:20.145225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:20.145236 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:20.145260 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:20.145272 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:20.145357 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:20.145420 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 14:40:20.145453 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 14:40:20.145717 24483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:20.146605 24483 log.cc:824] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:20.148248 24483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365: No bootstrap required, opened a new log
I20260923 14:40:20.151368 24483 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0ced8835f14ccfb75d9ec5705fd365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260923 14:40:20.151577 24483 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:20.151666 24483 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0ced8835f14ccfb75d9ec5705fd365, State: Initialized, Role: FOLLOWER
I20260923 14:40:20.152100 24483 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0ced8835f14ccfb75d9ec5705fd365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260923 14:40:20.152253 24483 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:20.152320 24483 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:20.152410 24483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:20.153221 24483 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0ced8835f14ccfb75d9ec5705fd365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260923 14:40:20.153523 24483 leader_election.cc:302] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee0ced8835f14ccfb75d9ec5705fd365; no voters:
I20260923 14:40:20.153785 24483 leader_election.cc:288] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:20.153920 24493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:20.154147 24493 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [term 1 LEADER]: Becoming Leader. State: Replica: ee0ced8835f14ccfb75d9ec5705fd365, State: Running, Role: LEADER
I20260923 14:40:20.154459 24493 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0ced8835f14ccfb75d9ec5705fd365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260923 14:40:20.154901 24483 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:20.156001 24495 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee0ced8835f14ccfb75d9ec5705fd365. Latest consensus state: current_term: 1 leader_uuid: "ee0ced8835f14ccfb75d9ec5705fd365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0ced8835f14ccfb75d9ec5705fd365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } } }
I20260923 14:40:20.156131 24495 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:20.156417 24494 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee0ced8835f14ccfb75d9ec5705fd365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee0ced8835f14ccfb75d9ec5705fd365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } } }
I20260923 14:40:20.156505 24494 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:20.156746 24502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:20.159510 24502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:20.164594 24502 catalog_manager.cc:1347] Generated new cluster ID: 1978fb00386244469425c64eb5bb5b22
I20260923 14:40:20.164654 24502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:20.171604 24502 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:20.171888 24502 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:20.178139 24502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee0ced8835f14ccfb75d9ec5705fd365: Generated new TSK 0
I20260923 14:40:20.179129 24502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:20.183921 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 24415
2026-09-23T14:40:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:20 dist-test-slave-zl47 krb5kdc[24514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:20 dist-test-slave-zl47 krb5kdc[24514](info): set up 2 sockets
Sep 23 14:40:20 dist-test-slave-zl47 krb5kdc[24514](info): commencing operation
krb5kdc: starting...
W20260923 14:40:21.106386 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.902s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:21 dist-test-slave-zl47 krb5kdc[24514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:40:21.206845 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:41403
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:41403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:21.298215 24527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:21.298432 24527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:21.298493 24527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:21.301836 24527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:21.301916 24527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:21.301955 24527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:21.301986 24527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:21.302017 24527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:21.306216 24527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:41403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:41403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:21.306967 24527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:21.307721 24527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:21.314569 24532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:21.314568 24533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:21.314577 24535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:21.314620 24527 server_base.cc:1034] running on GCE node
I20260923 14:40:21.315181 24527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:21.316000 24527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:21.317186 24527 hybrid_clock.cc:648] HybridClock initialized: now 1790174421317173 us; error 29 us; skew 500 ppm
Sep 23 14:40:21 dist-test-slave-zl47 krb5kdc[24514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174421, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:21.319497 24527 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:40:21.320286 24527 webserver.cc:460] Webserver started at http://127.2.218.62:43355/ using document root <none> and password file <none>
I20260923 14:40:21.320724 24527 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:21.320799 24527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:21.320968 24527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:21.322633 24527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "9333f2b9b5c146d390d61c24711a2904"
format_stamp: "Formatted at 2026-09-23 14:40:21 on dist-test-slave-zl47"
server_key: "cb09b47a083370a830b314bfc2825d43"
server_key_iv: "d00d716c2149a199cc6b13ca3c040cf8"
server_key_version: "encryptionkey@0"
I20260923 14:40:21.323086 24527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "9333f2b9b5c146d390d61c24711a2904"
format_stamp: "Formatted at 2026-09-23 14:40:21 on dist-test-slave-zl47"
server_key: "cb09b47a083370a830b314bfc2825d43"
server_key_iv: "d00d716c2149a199cc6b13ca3c040cf8"
server_key_version: "encryptionkey@0"
I20260923 14:40:21.325929 24527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:40:21.327797 24542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:21.328749 24527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:21.328876 24527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "9333f2b9b5c146d390d61c24711a2904"
format_stamp: "Formatted at 2026-09-23 14:40:21 on dist-test-slave-zl47"
server_key: "cb09b47a083370a830b314bfc2825d43"
server_key_iv: "d00d716c2149a199cc6b13ca3c040cf8"
server_key_version: "encryptionkey@0"
I20260923 14:40:21.328966 24527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:21.347456 24527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:21.349961 24527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:21.350173 24527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:21.355880 24527 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:41403
I20260923 14:40:21.355887 24594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:41403 every 8 connection(s)
I20260923 14:40:21.356570 24527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:21.358755 24595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:21.360859 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 24527
I20260923 14:40:21.360989 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:21.361232 2920 external_mini_cluster.cc:1428] Setting key e1239e5022195a821a993e95e8a87769
I20260923 14:40:21.363310 24595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904: Bootstrap starting.
I20260923 14:40:21.365514 24595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:21.366173 24595 log.cc:824] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:21.367635 24595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904: No bootstrap required, opened a new log
Sep 23 14:40:21 dist-test-slave-zl47 krb5kdc[24514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:21.371331 24595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9333f2b9b5c146d390d61c24711a2904" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41403 } }
I20260923 14:40:21.371565 24595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:21.371645 24595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9333f2b9b5c146d390d61c24711a2904, State: Initialized, Role: FOLLOWER
I20260923 14:40:21.372092 24595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9333f2b9b5c146d390d61c24711a2904" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41403 } }
I20260923 14:40:21.372201 24595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:21.372278 24595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:21.372363 24595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:21.373230 24595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9333f2b9b5c146d390d61c24711a2904" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41403 } }
I20260923 14:40:21.373564 24595 leader_election.cc:302] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9333f2b9b5c146d390d61c24711a2904; no voters:
I20260923 14:40:21.373791 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:21.362335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32971 (local address 127.2.218.62:41403)
0923 14:40:21.362662 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:21.362674 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:21.362702 (+ 28us) server_negotiation.cc:407] Connection header received
0923 14:40:21.363186 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:21.363198 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:21.363495 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:21.363714 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:21.364594 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:21.365516 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:21.366325 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:21.366889 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:21.367032 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:21.369360 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:21.369364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:21.369375 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:21.369396 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:21.370981 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:21.371529 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:21.371533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:21.371536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:21.371740 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:21.372035 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:21.372037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:21.372038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:21.372439 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:21.372595 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:21.372867 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 14:40:21.373072 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":114,"threads_started":1}
I20260923 14:40:21.373867 24595 leader_election.cc:288] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:21.373931 24600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:21.374164 24600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [term 1 LEADER]: Becoming Leader. State: Replica: 9333f2b9b5c146d390d61c24711a2904, State: Running, Role: LEADER
I20260923 14:40:21.374468 24600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9333f2b9b5c146d390d61c24711a2904" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41403 } }
I20260923 14:40:21.374971 24595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:21.375442 24600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9333f2b9b5c146d390d61c24711a2904. Latest consensus state: current_term: 1 leader_uuid: "9333f2b9b5c146d390d61c24711a2904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9333f2b9b5c146d390d61c24711a2904" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41403 } } }
I20260923 14:40:21.375586 24600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:21.375808 24601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9333f2b9b5c146d390d61c24711a2904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9333f2b9b5c146d390d61c24711a2904" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 41403 } } }
I20260923 14:40:21.375900 24601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:21.375932 24607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:21.377076 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:21.378469 24607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:21.383165 24607 catalog_manager.cc:1347] Generated new cluster ID: 299eb4df73294c6698832c155290975f
I20260923 14:40:21.383209 24607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:21.388975 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:21.378975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59095 (local address 127.2.218.62:41403)
0923 14:40:21.379068 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:21.379072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:21.381992 (+ 2920us) server_negotiation.cc:407] Connection header received
0923 14:40:21.382082 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:21.382087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:21.382149 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:21.382233 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:21.382773 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:21.383367 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:21.384405 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:21.384690 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:21.384740 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:21.386077 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:21.386080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:21.386082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:21.386122 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:21.387454 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:21.387875 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:21.387879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:21.387880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:21.387941 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:21.388215 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:21.388218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:21.388219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:21.388360 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:21.388455 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:21.388774 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 14:40:21.388850 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 14:40:21.390866 24607 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:21.391515 24607 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:21.410437 24607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9333f2b9b5c146d390d61c24711a2904: Generated new TSK 0
I20260923 14:40:21.410940 24607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:21.421531 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 24527
W20260923 14:40:21.426052 24512 connection.cc:535] client connection to 127.2.218.62:41403 recv error: Network error: failed to read from TLS socket (remote: 127.2.218.62:41403): Connection reset by peer (error 104)
2026-09-23T14:40:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1235 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:21 dist-test-slave-zl47 krb5kdc[24626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:21 dist-test-slave-zl47 krb5kdc[24626](info): set up 2 sockets
Sep 23 14:40:21 dist-test-slave-zl47 krb5kdc[24626](info): commencing operation
krb5kdc: starting...
W20260923 14:40:22.310691 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.871s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:22 dist-test-slave-zl47 krb5kdc[24626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:40:22.409979 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:43921
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:22.504235 24639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:22.504463 24639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:22.504527 24639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:22.507869 24639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:22.507946 24639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:22.507982 24639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:22.508016 24639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:22.508049 24639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:22.512286 24639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:43921
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:22.513089 24639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:22.513891 24639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:40:22.520866 24639 server_base.cc:1034] running on GCE node
W20260923 14:40:22.520908 24644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:22.521034 24647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:22.520924 24645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:22.521615 24639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:22.522697 24639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:22.523875 24639 hybrid_clock.cc:648] HybridClock initialized: now 1790174422523859 us; error 27 us; skew 500 ppm
Sep 23 14:40:22 dist-test-slave-zl47 krb5kdc[24626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174422, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:22.526412 24639 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:40:22.527091 24639 webserver.cc:460] Webserver started at http://127.2.218.62:38809/ using document root <none> and password file <none>
I20260923 14:40:22.527549 24639 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:22.527628 24639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:22.527818 24639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:22.529592 24639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "fef333f0c38c46c882bc3dde527cf656"
format_stamp: "Formatted at 2026-09-23 14:40:22 on dist-test-slave-zl47"
server_key: "9a62c829ccfa3bbb01471eee760457e9"
server_key_iv: "96e37f54c6a54cc9478b250bc754052b"
server_key_version: "encryptionkey@0"
I20260923 14:40:22.530161 24639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "fef333f0c38c46c882bc3dde527cf656"
format_stamp: "Formatted at 2026-09-23 14:40:22 on dist-test-slave-zl47"
server_key: "9a62c829ccfa3bbb01471eee760457e9"
server_key_iv: "96e37f54c6a54cc9478b250bc754052b"
server_key_version: "encryptionkey@0"
I20260923 14:40:22.533102 24639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 14:40:22.535061 24654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:22.536051 24639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:40:22.536186 24639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "fef333f0c38c46c882bc3dde527cf656"
format_stamp: "Formatted at 2026-09-23 14:40:22 on dist-test-slave-zl47"
server_key: "9a62c829ccfa3bbb01471eee760457e9"
server_key_iv: "96e37f54c6a54cc9478b250bc754052b"
server_key_version: "encryptionkey@0"
I20260923 14:40:22.536278 24639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:22.553378 24639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:22.555608 24639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:22.555771 24639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:22.561391 24639 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:43921
I20260923 14:40:22.561396 24706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:43921 every 8 connection(s)
I20260923 14:40:22.562157 24639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:22.564034 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 24639
I20260923 14:40:22.564148 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:22.564391 2920 external_mini_cluster.cc:1428] Setting key b048e203e6d011912b6d34c45c2e7dc3
I20260923 14:40:22.564531 24707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:22.570042 24707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656: Bootstrap starting.
I20260923 14:40:22.571856 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:22.565435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40787 (local address 127.2.218.62:43921)
0923 14:40:22.565769 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:22.565781 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:22.565809 (+ 28us) server_negotiation.cc:407] Connection header received
0923 14:40:22.566667 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:22.566683 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:22.566943 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:22.567165 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:22.567469 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:22.568243 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:22.569318 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:22.569713 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:22.569817 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:22.570554 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:22.570558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:22.570567 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:22.570850 (+ 283us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:22.570892 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:22.570910 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:22.570994 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:22.571157 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 14:40:22.571293 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":118,"threads_started":1}
I20260923 14:40:22.571964 24707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:22.572561 24707 log.cc:824] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:22.574259 24707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656: No bootstrap required, opened a new log
I20260923 14:40:22.574421 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:22.578037 24707 raft_consensus.cc:348] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef333f0c38c46c882bc3dde527cf656" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43921 } }
I20260923 14:40:22.578274 24707 raft_consensus.cc:374] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:22.578341 24707 raft_consensus.cc:729] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef333f0c38c46c882bc3dde527cf656, State: Initialized, Role: FOLLOWER
I20260923 14:40:22.578706 24707 consensus_queue.cc:260] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef333f0c38c46c882bc3dde527cf656" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43921 } }
I20260923 14:40:22.578852 24707 raft_consensus.cc:388] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:22.578945 24707 raft_consensus.cc:482] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:22.579051 24707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:22.579706 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:22.576281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57193 (local address 127.2.218.62:43921)
0923 14:40:22.576426 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:22.576428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:22.576529 (+ 101us) server_negotiation.cc:407] Connection header received
0923 14:40:22.576617 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:22.576618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:22.576644 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:22.576695 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:22.576898 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:22.577286 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:22.578229 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:22.578539 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:22.578628 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:22.579325 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:22.579328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:22.579330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:22.579340 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:22.579360 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:22.579371 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:22.579461 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:22.579562 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 14:40:22.579587 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 14:40:22.579695 24707 raft_consensus.cc:504] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef333f0c38c46c882bc3dde527cf656" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43921 } }
I20260923 14:40:22.580013 24707 leader_election.cc:302] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fef333f0c38c46c882bc3dde527cf656; no voters:
I20260923 14:40:22.580274 24707 leader_election.cc:288] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:22.580331 24717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:22.580514 24717 raft_consensus.cc:686] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [term 1 LEADER]: Becoming Leader. State: Replica: fef333f0c38c46c882bc3dde527cf656, State: Running, Role: LEADER
I20260923 14:40:22.580780 24717 consensus_queue.cc:237] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef333f0c38c46c882bc3dde527cf656" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43921 } }
I20260923 14:40:22.581278 24707 sys_catalog.cc:564] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:22.582204 24719 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fef333f0c38c46c882bc3dde527cf656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef333f0c38c46c882bc3dde527cf656" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43921 } } }
I20260923 14:40:22.582340 24719 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:22.582592 24718 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fef333f0c38c46c882bc3dde527cf656. Latest consensus state: current_term: 1 leader_uuid: "fef333f0c38c46c882bc3dde527cf656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef333f0c38c46c882bc3dde527cf656" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 43921 } } }
I20260923 14:40:22.582695 24718 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:22.582721 24726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:22.585003 24726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:22.589519 24726 catalog_manager.cc:1347] Generated new cluster ID: b3dea0e1d8ad46b0a9f097afc561e1c3
I20260923 14:40:22.589570 24726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:22.595518 24726 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:22.596530 24726 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:22.601086 24726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fef333f0c38c46c882bc3dde527cf656: Generated new TSK 0
I20260923 14:40:22.601642 24726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:22.608094 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 24639
W20260923 14:40:22.612840 24624 connection.cc:535] client connection to 127.2.218.62:43921 recv error: Network error: failed to read from TLS socket (remote: 127.2.218.62:43921): Connection reset by peer (error 104)
2026-09-23T14:40:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1186 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:22 dist-test-slave-zl47 krb5kdc[24738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:22 dist-test-slave-zl47 krb5kdc[24738](info): set up 2 sockets
Sep 23 14:40:22 dist-test-slave-zl47 krb5kdc[24738](info): commencing operation
krb5kdc: starting...
W20260923 14:40:23.514312 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.888s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:23 dist-test-slave-zl47 krb5kdc[24738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:40:23.616472 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36401
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:36401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:23.710223 24751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:23.710423 24751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:23.710460 24751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:23.713768 24751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:23.713819 24751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:23.713837 24751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:23.713896 24751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:23.713927 24751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:23.718312 24751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:36401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:23.719120 24751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:23.719929 24751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:23.727411 24757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:23.727468 24756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:23.727440 24759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:23.727571 24751 server_base.cc:1034] running on GCE node
I20260923 14:40:23.728034 24751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:23.728837 24751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:23.729997 24751 hybrid_clock.cc:648] HybridClock initialized: now 1790174423729977 us; error 31 us; skew 500 ppm
Sep 23 14:40:23 dist-test-slave-zl47 krb5kdc[24738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174423, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:23.732270 24751 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:40:23.733040 24751 webserver.cc:460] Webserver started at http://127.2.218.62:45945/ using document root <none> and password file <none>
I20260923 14:40:23.733495 24751 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:23.733577 24751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:23.733798 24751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:23.735550 24751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "6cbd9a409039431f8fe4417bbeeaa182"
format_stamp: "Formatted at 2026-09-23 14:40:23 on dist-test-slave-zl47"
server_key: "eeea5bfb7c983aa5137b534125d380a7"
server_key_iv: "6982ddc809890d3b0ee61d4f72213783"
server_key_version: "encryptionkey@0"
I20260923 14:40:23.736066 24751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "6cbd9a409039431f8fe4417bbeeaa182"
format_stamp: "Formatted at 2026-09-23 14:40:23 on dist-test-slave-zl47"
server_key: "eeea5bfb7c983aa5137b534125d380a7"
server_key_iv: "6982ddc809890d3b0ee61d4f72213783"
server_key_version: "encryptionkey@0"
I20260923 14:40:23.739342 24751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:40:23.741575 24766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:23.742596 24751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260923 14:40:23.742725 24751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "6cbd9a409039431f8fe4417bbeeaa182"
format_stamp: "Formatted at 2026-09-23 14:40:23 on dist-test-slave-zl47"
server_key: "eeea5bfb7c983aa5137b534125d380a7"
server_key_iv: "6982ddc809890d3b0ee61d4f72213783"
server_key_version: "encryptionkey@0"
I20260923 14:40:23.742826 24751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:23.746701 24751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:23.749034 24751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:23.749212 24751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:23.755338 24751 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:36401
I20260923 14:40:23.755347 24818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:36401 every 8 connection(s)
I20260923 14:40:23.756096 24751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:23.758383 24819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:23.760578 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 24751
I20260923 14:40:23.760725 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:23.760977 2920 external_mini_cluster.cc:1428] Setting key c4c071d156b2108f3951796b0ff9aa8d
I20260923 14:40:23.763837 24819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182: Bootstrap starting.
I20260923 14:40:23.765667 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:23.762211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58727 (local address 127.2.218.62:36401)
0923 14:40:23.762573 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:23.762588 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:23.762623 (+ 35us) server_negotiation.cc:407] Connection header received
0923 14:40:23.763280 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:23.763300 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:23.763594 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:23.763895 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:23.764542 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:23.764547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:23.764558 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:23.764569 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:23.764621 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:23.764643 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:23.764766 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:23.764836 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 14:40:23.764966 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
I20260923 14:40:23.766078 24819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:23.766806 24819 log.cc:824] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:23.768126 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:23.768519 24819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182: No bootstrap required, opened a new log
I20260923 14:40:23.771147 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:23.769844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49347 (local address 127.2.218.62:36401)
0923 14:40:23.769935 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:23.769938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:23.770048 (+ 110us) server_negotiation.cc:407] Connection header received
0923 14:40:23.770129 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:23.770131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:23.770159 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:23.770230 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:23.770793 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:23.770797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:23.770800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:23.770815 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:23.770844 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:23.770857 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:23.770959 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:23.770989 (+ 30us) server_negotiation.cc:299] Negotiation successful
0923 14:40:23.771021 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:40:23.771932 24819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbd9a409039431f8fe4417bbeeaa182" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36401 } }
I20260923 14:40:23.772105 24819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:23.772135 24819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cbd9a409039431f8fe4417bbeeaa182, State: Initialized, Role: FOLLOWER
I20260923 14:40:23.772428 24819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbd9a409039431f8fe4417bbeeaa182" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36401 } }
I20260923 14:40:23.772521 24819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:23.772562 24819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:23.772626 24819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:23.773301 24819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbd9a409039431f8fe4417bbeeaa182" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36401 } }
I20260923 14:40:23.773602 24819 leader_election.cc:302] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cbd9a409039431f8fe4417bbeeaa182; no voters:
I20260923 14:40:23.773854 24819 leader_election.cc:288] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:23.773918 24829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:23.774128 24829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [term 1 LEADER]: Becoming Leader. State: Replica: 6cbd9a409039431f8fe4417bbeeaa182, State: Running, Role: LEADER
I20260923 14:40:23.774396 24829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbd9a409039431f8fe4417bbeeaa182" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36401 } }
I20260923 14:40:23.775004 24819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:23.776043 24830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cbd9a409039431f8fe4417bbeeaa182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbd9a409039431f8fe4417bbeeaa182" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36401 } } }
I20260923 14:40:23.776050 24831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cbd9a409039431f8fe4417bbeeaa182. Latest consensus state: current_term: 1 leader_uuid: "6cbd9a409039431f8fe4417bbeeaa182" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cbd9a409039431f8fe4417bbeeaa182" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36401 } } }
I20260923 14:40:23.776235 24831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:23.776227 24830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:23.776719 24838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:23.779063 24838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:23.783208 24838 catalog_manager.cc:1347] Generated new cluster ID: 85dfc4f6d69d4985b64893a0d348c2cb
I20260923 14:40:23.783258 24838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:23.792172 24838 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:23.792428 24838 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:23.800431 24838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cbd9a409039431f8fe4417bbeeaa182: Generated new TSK 0
I20260923 14:40:23.800899 24838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:23.805490 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 24751
2026-09-23T14:40:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:23 dist-test-slave-zl47 krb5kdc[24850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:23 dist-test-slave-zl47 krb5kdc[24850](info): set up 2 sockets
Sep 23 14:40:23 dist-test-slave-zl47 krb5kdc[24850](info): commencing operation
krb5kdc: starting...
W20260923 14:40:24.702500 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.879s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:24 dist-test-slave-zl47 krb5kdc[24850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:40:24.806512 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:35279
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:35279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:24.897178 24863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:24.897428 24863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:24.897493 24863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:24.900826 24863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:24.900906 24863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:24.900938 24863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:24.900986 24863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:24.901019 24863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:24.905648 24863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:35279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:35279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:24.906497 24863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:24.907328 24863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:24.914537 24871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:24.914548 24868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:24.914544 24869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:24.914615 24863 server_base.cc:1034] running on GCE node
I20260923 14:40:24.915177 24863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:24.916026 24863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:24.917204 24863 hybrid_clock.cc:648] HybridClock initialized: now 1790174424917182 us; error 35 us; skew 500 ppm
Sep 23 14:40:24 dist-test-slave-zl47 krb5kdc[24850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174424, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:24.919761 24863 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:40:24.920563 24863 webserver.cc:460] Webserver started at http://127.2.218.62:40163/ using document root <none> and password file <none>
I20260923 14:40:24.921067 24863 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:24.921170 24863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:24.921404 24863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:24.923178 24863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "60ff74eb08e842d4980be9474ab03009"
format_stamp: "Formatted at 2026-09-23 14:40:24 on dist-test-slave-zl47"
server_key: "c05f82afe2d00896ae8c9587cb954cd7"
server_key_iv: "defd9a4553b47135c7df491d426c91ce"
server_key_version: "encryptionkey@0"
I20260923 14:40:24.923656 24863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "60ff74eb08e842d4980be9474ab03009"
format_stamp: "Formatted at 2026-09-23 14:40:24 on dist-test-slave-zl47"
server_key: "c05f82afe2d00896ae8c9587cb954cd7"
server_key_iv: "defd9a4553b47135c7df491d426c91ce"
server_key_version: "encryptionkey@0"
I20260923 14:40:24.926972 24863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:40:24.928927 24878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:24.929890 24863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:24.930009 24863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "60ff74eb08e842d4980be9474ab03009"
format_stamp: "Formatted at 2026-09-23 14:40:24 on dist-test-slave-zl47"
server_key: "c05f82afe2d00896ae8c9587cb954cd7"
server_key_iv: "defd9a4553b47135c7df491d426c91ce"
server_key_version: "encryptionkey@0"
I20260923 14:40:24.930099 24863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:24.946645 24863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:24.949102 24863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:24.949319 24863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:24.955411 24863 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:35279
I20260923 14:40:24.955438 24930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:35279 every 8 connection(s)
I20260923 14:40:24.956220 24863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:24.958587 24931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:24.960708 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 24863
I20260923 14:40:24.960826 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:24.961043 2920 external_mini_cluster.cc:1428] Setting key ea75a885c8fa22bc84a6bfade1bf66fd
I20260923 14:40:24.964047 24931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009: Bootstrap starting.
I20260923 14:40:24.966324 24931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:24.967072 24931 log.cc:824] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:24.968566 24931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009: No bootstrap required, opened a new log
Sep 23 14:40:24 dist-test-slave-zl47 krb5kdc[24850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:24.971278 24931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff74eb08e842d4980be9474ab03009" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35279 } }
I20260923 14:40:24.971478 24931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:24.971549 24931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ff74eb08e842d4980be9474ab03009, State: Initialized, Role: FOLLOWER
I20260923 14:40:24.972011 24931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff74eb08e842d4980be9474ab03009" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35279 } }
I20260923 14:40:24.972143 24931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:24.972219 24931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:24.972302 24931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:24.972982 24931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff74eb08e842d4980be9474ab03009" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35279 } }
I20260923 14:40:24.973268 24931 leader_election.cc:302] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ff74eb08e842d4980be9474ab03009; no voters:
I20260923 14:40:24.973608 24931 leader_election.cc:288] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:24.973667 24936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:24.973906 24936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [term 1 LEADER]: Becoming Leader. State: Replica: 60ff74eb08e842d4980be9474ab03009, State: Running, Role: LEADER
I20260923 14:40:24.974258 24936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff74eb08e842d4980be9474ab03009" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35279 } }
I20260923 14:40:24.974817 24931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:24.974925 24934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:24.962214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51463 (local address 127.2.218.62:35279)
0923 14:40:24.962580 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:24.962597 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:24.962634 (+ 37us) server_negotiation.cc:407] Connection header received
0923 14:40:24.963375 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:24.963395 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:24.963738 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:24.964058 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:24.964701 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:24.965957 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:24.966779 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:24.967399 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:24.967571 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:24.969817 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:24.969822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:24.969832 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:24.969861 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:24.971904 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:24.972348 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:24.972354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:24.972359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:24.972503 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:24.972779 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:24.972783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:24.972785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:24.973164 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:24.973391 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:24.973708 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 14:40:24.974023 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
I20260923 14:40:24.975342 24938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ff74eb08e842d4980be9474ab03009. Latest consensus state: current_term: 1 leader_uuid: "60ff74eb08e842d4980be9474ab03009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff74eb08e842d4980be9474ab03009" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35279 } } }
I20260923 14:40:24.975718 24937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60ff74eb08e842d4980be9474ab03009" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff74eb08e842d4980be9474ab03009" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35279 } } }
I20260923 14:40:24.975811 24937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:24.976636 24938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:24.977121 24940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:24.977995 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:24.979838 24940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:24.984999 24940 catalog_manager.cc:1347] Generated new cluster ID: 001cd91bd23b4022b0edfb4b8cdd0fe9
I20260923 14:40:24.985052 24940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:24.989943 24934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:24.982530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47593 (local address 127.2.218.62:35279)
0923 14:40:24.982617 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:24.982621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:24.982840 (+ 219us) server_negotiation.cc:407] Connection header received
0923 14:40:24.982922 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:24.982925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:24.982970 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:24.983053 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:24.983729 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:24.984186 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:24.985019 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:24.985331 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:24.985452 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:24.986819 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:24.986822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:24.986825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:24.986847 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:24.988385 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:24.988887 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:24.988890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:24.988891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:24.988965 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:24.989217 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:24.989220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:24.989221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:24.989397 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:24.989499 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:24.989717 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 14:40:24.989810 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 14:40:25.012017 24940 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:25.013108 24940 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:25.020828 24940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60ff74eb08e842d4980be9474ab03009: Generated new TSK 0
I20260923 14:40:25.021493 24940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:25.027473 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 24863
2026-09-23T14:40:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:25 dist-test-slave-zl47 krb5kdc[24962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:25 dist-test-slave-zl47 krb5kdc[24962](info): set up 2 sockets
Sep 23 14:40:25 dist-test-slave-zl47 krb5kdc[24962](info): commencing operation
krb5kdc: starting...
W20260923 14:40:25.930619 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.885s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:25 dist-test-slave-zl47 krb5kdc[24962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:40:26.034976 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:35061
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:26.135506 24975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:26.135746 24975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:26.135785 24975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:26.139356 24975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:26.139410 24975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:26.139429 24975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:26.139472 24975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:26.139508 24975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:26.143909 24975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:26.145058 24975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:26.145951 24975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:26.153314 24980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:26.153602 24975 server_base.cc:1034] running on GCE node
W20260923 14:40:26.153317 24983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:26.153335 24981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:26.154214 24975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:26.155047 24975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:26.156198 24975 hybrid_clock.cc:648] HybridClock initialized: now 1790174426156182 us; error 28 us; skew 500 ppm
Sep 23 14:40:26 dist-test-slave-zl47 krb5kdc[24962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:26.158609 24975 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:40:26.159303 24975 webserver.cc:460] Webserver started at http://127.2.218.62:44499/ using document root <none> and password file <none>
I20260923 14:40:26.159819 24975 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:26.159898 24975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:26.160074 24975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:26.161780 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "2f6817ece852488aabf383ba53b386f0"
format_stamp: "Formatted at 2026-09-23 14:40:26 on dist-test-slave-zl47"
server_key: "5bbff8ae62b24473cfa0ecbccd35b1f5"
server_key_iv: "a62b772babf36653218532e24ac5a545"
server_key_version: "encryptionkey@0"
I20260923 14:40:26.162348 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "2f6817ece852488aabf383ba53b386f0"
format_stamp: "Formatted at 2026-09-23 14:40:26 on dist-test-slave-zl47"
server_key: "5bbff8ae62b24473cfa0ecbccd35b1f5"
server_key_iv: "a62b772babf36653218532e24ac5a545"
server_key_version: "encryptionkey@0"
I20260923 14:40:26.165632 24975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:40:26.167755 24990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:26.168864 24975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:40:26.168994 24975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "2f6817ece852488aabf383ba53b386f0"
format_stamp: "Formatted at 2026-09-23 14:40:26 on dist-test-slave-zl47"
server_key: "5bbff8ae62b24473cfa0ecbccd35b1f5"
server_key_iv: "a62b772babf36653218532e24ac5a545"
server_key_version: "encryptionkey@0"
I20260923 14:40:26.169094 24975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:26.179355 24975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:26.181803 24975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:26.181970 24975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:26.187971 24975 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:35061
I20260923 14:40:26.187992 25042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:35061 every 8 connection(s)
I20260923 14:40:26.188714 24975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:26.190310 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 24975
I20260923 14:40:26.190431 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:26.190659 2920 external_mini_cluster.cc:1428] Setting key 7195d28448986e59e58ac696e71f9bdf
I20260923 14:40:26.191134 25043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:26.196381 25043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0: Bootstrap starting.
I20260923 14:40:26.197913 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:26.191712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48929 (local address 127.2.218.62:35061)
0923 14:40:26.192028 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:26.192057 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:26.192086 (+ 29us) server_negotiation.cc:407] Connection header received
0923 14:40:26.192593 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:26.192606 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:26.192862 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:26.193087 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:26.193423 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:26.194287 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:26.195337 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:26.195716 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:26.195811 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:26.196561 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:26.196565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:26.196574 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:26.196871 (+ 297us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:26.196917 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:26.196935 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:26.197006 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:26.197149 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 14:40:26.197324 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":107,"threads_started":1}
I20260923 14:40:26.198305 25043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:26.198930 25043 log.cc:824] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:26.200306 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:26.200636 25043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0: No bootstrap required, opened a new log
I20260923 14:40:26.204164 25043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6817ece852488aabf383ba53b386f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35061 } }
I20260923 14:40:26.204382 25043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:26.204437 25043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6817ece852488aabf383ba53b386f0, State: Initialized, Role: FOLLOWER
I20260923 14:40:26.204815 25043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6817ece852488aabf383ba53b386f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35061 } }
I20260923 14:40:26.204957 25043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:26.205049 25043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:26.205152 25043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:26.205644 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:26.202061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44723 (local address 127.2.218.62:35061)
0923 14:40:26.202183 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:26.202187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:26.202291 (+ 104us) server_negotiation.cc:407] Connection header received
0923 14:40:26.202364 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:26.202367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:26.202406 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:26.202476 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:26.202685 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:26.203460 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:26.204121 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:26.204418 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:26.204512 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:26.205252 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:26.205256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:26.205261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:26.205272 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:26.205295 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:26.205305 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:26.205362 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:26.205507 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 14:40:26.205540 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 14:40:26.206174 25043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6817ece852488aabf383ba53b386f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35061 } }
I20260923 14:40:26.206492 25043 leader_election.cc:302] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f6817ece852488aabf383ba53b386f0; no voters:
I20260923 14:40:26.206779 25043 leader_election.cc:288] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:26.207105 25054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:26.207286 25054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [term 1 LEADER]: Becoming Leader. State: Replica: 2f6817ece852488aabf383ba53b386f0, State: Running, Role: LEADER
I20260923 14:40:26.207554 25054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6817ece852488aabf383ba53b386f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35061 } }
I20260923 14:40:26.207777 25043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:26.208972 25055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f6817ece852488aabf383ba53b386f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6817ece852488aabf383ba53b386f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35061 } } }
I20260923 14:40:26.209075 25055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:26.209606 25063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:26.210502 25053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f6817ece852488aabf383ba53b386f0. Latest consensus state: current_term: 1 leader_uuid: "2f6817ece852488aabf383ba53b386f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6817ece852488aabf383ba53b386f0" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 35061 } } }
I20260923 14:40:26.210594 25053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:26.212152 25063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:26.217298 25063 catalog_manager.cc:1347] Generated new cluster ID: 1ddf7ee05fde4de09f12694286bd4e3c
I20260923 14:40:26.217355 25063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:26.230981 25063 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:26.231731 25063 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:26.234586 25063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f6817ece852488aabf383ba53b386f0: Generated new TSK 0
I20260923 14:40:26.235117 25063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:26.238711 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 24975
2026-09-23T14:40:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:26 dist-test-slave-zl47 krb5kdc[25074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:26 dist-test-slave-zl47 krb5kdc[25074](info): set up 2 sockets
Sep 23 14:40:26 dist-test-slave-zl47 krb5kdc[25074](info): commencing operation
krb5kdc: starting...
W20260923 14:40:27.146842 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.890s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:27 dist-test-slave-zl47 krb5kdc[25074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:40:27.257091 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36911
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:36911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:27.355540 25087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:27.355731 25087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:27.355767 25087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:27.359608 25087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:27.359654 25087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:27.359673 25087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:27.359687 25087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:27.359701 25087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:27.364418 25087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:36911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:36911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:27.365217 25087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:27.366058 25087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:27.373562 25092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:27.373590 25093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:27.373633 25095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:27.373806 25087 server_base.cc:1034] running on GCE node
I20260923 14:40:27.374426 25087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:27.375326 25087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:27.376487 25087 hybrid_clock.cc:648] HybridClock initialized: now 1790174427376469 us; error 28 us; skew 500 ppm
Sep 23 14:40:27 dist-test-slave-zl47 krb5kdc[25074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174427, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:27.379072 25087 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:40:27.379891 25087 webserver.cc:460] Webserver started at http://127.2.218.62:36269/ using document root <none> and password file <none>
I20260923 14:40:27.380389 25087 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:27.380483 25087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:27.380692 25087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:27.382534 25087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "ca99396afdea40a496cca9952c269594"
format_stamp: "Formatted at 2026-09-23 14:40:27 on dist-test-slave-zl47"
server_key: "a74eac789ff0275a5a18540aac1f89ea"
server_key_iv: "3aa2f44e406044b85845bf1a04d4b94c"
server_key_version: "encryptionkey@0"
I20260923 14:40:27.383025 25087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "ca99396afdea40a496cca9952c269594"
format_stamp: "Formatted at 2026-09-23 14:40:27 on dist-test-slave-zl47"
server_key: "a74eac789ff0275a5a18540aac1f89ea"
server_key_iv: "3aa2f44e406044b85845bf1a04d4b94c"
server_key_version: "encryptionkey@0"
I20260923 14:40:27.386338 25087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260923 14:40:27.388453 25102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:27.389461 25087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:27.389602 25087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "ca99396afdea40a496cca9952c269594"
format_stamp: "Formatted at 2026-09-23 14:40:27 on dist-test-slave-zl47"
server_key: "a74eac789ff0275a5a18540aac1f89ea"
server_key_iv: "3aa2f44e406044b85845bf1a04d4b94c"
server_key_version: "encryptionkey@0"
I20260923 14:40:27.389699 25087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:27.393812 25087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:27.396225 25087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:27.396409 25087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:27.403144 25087 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:36911
I20260923 14:40:27.403160 25154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:36911 every 8 connection(s)
I20260923 14:40:27.403873 25087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:27.406234 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:27.410957 25155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594: Bootstrap starting.
I20260923 14:40:27.411775 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 25087
I20260923 14:40:27.411906 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:27.412168 2920 external_mini_cluster.cc:1428] Setting key 8d648652b5da0d7070327e208635a3c0
I20260923 14:40:27.413403 25155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:27.414085 25155 log.cc:824] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:27.415871 25155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594: No bootstrap required, opened a new log
I20260923 14:40:27.416637 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:27.413165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34179 (local address 127.2.218.62:36911)
0923 14:40:27.413618 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:27.413631 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:27.413662 (+ 31us) server_negotiation.cc:407] Connection header received
0923 14:40:27.414344 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:27.414361 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:27.414616 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:27.414925 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:27.415610 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:27.415615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:27.415626 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:27.415635 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:27.415680 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:27.415697 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:27.415804 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:27.415895 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 14:40:27.416028 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":159,"threads_started":1}
I20260923 14:40:27.419042 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:27.419222 25155 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca99396afdea40a496cca9952c269594" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36911 } }
I20260923 14:40:27.419479 25155 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 14:40:27.419555 25155 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca99396afdea40a496cca9952c269594, State: Initialized, Role: FOLLOWER
I20260923 14:40:27.419946 25155 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca99396afdea40a496cca9952c269594" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36911 } }
I20260923 14:40:27.420089 25155 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 14:40:27.420172 25155 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 14:40:27.420284 25155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:27.421196 25155 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca99396afdea40a496cca9952c269594" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36911 } }
I20260923 14:40:27.421516 25155 leader_election.cc:302] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca99396afdea40a496cca9952c269594; no voters:
I20260923 14:40:27.421811 25155 leader_election.cc:288] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 14:40:27.421864 25165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:27.422030 25165 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [term 1 LEADER]: Becoming Leader. State: Replica: ca99396afdea40a496cca9952c269594, State: Running, Role: LEADER
I20260923 14:40:27.422300 25165 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca99396afdea40a496cca9952c269594" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36911 } }
I20260923 14:40:27.422915 25155 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:27.422937 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:27.421232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45905 (local address 127.2.218.62:36911)
0923 14:40:27.421349 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:27.421354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:27.421555 (+ 201us) server_negotiation.cc:407] Connection header received
0923 14:40:27.421680 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:27.421682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:27.421710 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:27.421793 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:27.422554 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:27.422558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:27.422561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 14:40:27.422573 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:27.422599 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 14:40:27.422612 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:27.422660 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:27.422803 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 14:40:27.422834 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:40:27.423343 25166 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca99396afdea40a496cca9952c269594. Latest consensus state: current_term: 1 leader_uuid: "ca99396afdea40a496cca9952c269594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca99396afdea40a496cca9952c269594" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36911 } } }
I20260923 14:40:27.423373 25167 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca99396afdea40a496cca9952c269594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca99396afdea40a496cca9952c269594" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 36911 } } }
I20260923 14:40:27.423455 25166 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:27.423466 25167 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594 [sys.catalog]: This master's current role is: LEADER
W20260923 14:40:27.427031 25180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:40:27.427126 25180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:40:27.427198 25170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:27.427788 25170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:27.433045 25170 catalog_manager.cc:1347] Generated new cluster ID: 2f39ee48348b40b88fe464bd02c7ffac
I20260923 14:40:27.433100 25170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:27.460551 25170 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:27.460841 25170 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:27.473980 25170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca99396afdea40a496cca9952c269594: Generated new TSK 0
I20260923 14:40:27.474712 25170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 14:40:27.486759 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 25087
2026-09-23T14:40:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1248 ms)
[----------] 18 tests from AuthTokenIssuingTest (21950 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:27 dist-test-slave-zl47 krb5kdc[25186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:27 dist-test-slave-zl47 krb5kdc[25186](info): set up 2 sockets
Sep 23 14:40:27 dist-test-slave-zl47 krb5kdc[25186](info): commencing operation
krb5kdc: starting...
W20260923 14:40:28.458626 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:28 dist-test-slave-zl47 krb5kdc[25186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:40:28.561138 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:42327
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:42327,127.2.218.61:36205,127.2.218.60:34205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:28.654233 25199 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:40:28.654488 25199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:28.654548 25199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:28.654609 25199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:28.657886 25199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:28.657960 25199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:28.657992 25199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:28.658023 25199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:28.658054 25199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:28.662386 25199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:42327,127.2.218.61:36205,127.2.218.60:34205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:42327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:28.663198 25199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:28.664032 25199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:28.671010 25207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:28.671089 25199 server_base.cc:1034] running on GCE node
W20260923 14:40:28.671028 25205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:28.671020 25204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:28.671749 25199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:28.672696 25199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:28.673885 25199 hybrid_clock.cc:648] HybridClock initialized: now 1790174428673864 us; error 36 us; skew 500 ppm
Sep 23 14:40:28 dist-test-slave-zl47 krb5kdc[25186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:28.681975 25199 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:40:28.682726 25199 webserver.cc:460] Webserver started at http://127.2.218.62:39617/ using document root <none> and password file <none>
I20260923 14:40:28.683176 25199 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:28.683259 25199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:28.683456 25199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:28.685112 25199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "f05bde479864490f8cf2646d47016741"
format_stamp: "Formatted at 2026-09-23 14:40:28 on dist-test-slave-zl47"
server_key: "3999061130ddd4887f13c0668f6353eb"
server_key_iv: "5582587056682f66d26d4e314cf788aa"
server_key_version: "encryptionkey@0"
I20260923 14:40:28.685611 25199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "f05bde479864490f8cf2646d47016741"
format_stamp: "Formatted at 2026-09-23 14:40:28 on dist-test-slave-zl47"
server_key: "3999061130ddd4887f13c0668f6353eb"
server_key_iv: "5582587056682f66d26d4e314cf788aa"
server_key_version: "encryptionkey@0"
I20260923 14:40:28.688707 25199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260923 14:40:28.690778 25214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:28.691783 25199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:28.691912 25199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "f05bde479864490f8cf2646d47016741"
format_stamp: "Formatted at 2026-09-23 14:40:28 on dist-test-slave-zl47"
server_key: "3999061130ddd4887f13c0668f6353eb"
server_key_iv: "5582587056682f66d26d4e314cf788aa"
server_key_version: "encryptionkey@0"
I20260923 14:40:28.692006 25199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:28.709182 25199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:28.711355 25199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:28.711531 25199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:28.717305 25199 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:42327
I20260923 14:40:28.717585 25266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:42327 every 8 connection(s)
I20260923 14:40:28.718169 25199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:28.720600 25267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:28.723785 25267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:40:28.725466 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 25199
I20260923 14:40:28.725612 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:28.725844 2920 external_mini_cluster.cc:1428] Setting key 13b32c3b1af7fea25539ea4ca54979c1
Sep 23 14:40:28 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.61@KRBTEST.COM" created.
I20260923 14:40:28.737027 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.725659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.2.218.62:42327)
0923 14:40:28.726087 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:28.726124 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:28.726354 (+ 230us) server_negotiation.cc:407] Connection header received
0923 14:40:28.727243 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:28.727255 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:28.727405 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:28.727577 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:28.728339 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.729129 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.730043 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.730422 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.730586 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:28.732834 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:28.732839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:28.732847 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:28.732865 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:28.734387 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:28.734799 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:28.734803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:28.734804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:28.734890 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:28.735295 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:28.735300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:28.735304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:28.735748 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:28.735866 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:28.736184 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 14:40:28.736386 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":300,"thread_start_us":130,"threads_started":1}
I20260923 14:40:28.737028 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.725659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:42327 (local address 127.0.0.1:46218)
0923 14:40:28.726202 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.726244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:28.726975 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:28.727575 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:28.727579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:28.727964 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:28.728190 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:28.728202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.729257 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.729261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:28.729925 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:28.729934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.730574 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.730578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:28.730682 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:28.731234 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:28.731249 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:28.732698 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:28.734497 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:28.734512 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:28.734518 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:28.734714 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:28.735089 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:28.735091 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:28.735093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:28.735224 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:28.735859 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:28.735863 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:28.735970 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 14:40:28.736090 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 14:40:28.736288 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"spinlock_wait_cycles":39296,"thread_start_us":135,"threads_started":1}
I20260923 14:40:28.739158 25267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:40:28.739748 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.739552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:36205 (local address 127.0.0.1:46496)
0923 14:40:28.739649 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.739704 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 14:40:28.739986 25215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.218.61:36205: connect: Connection refused (error 111)
I20260923 14:40:28.740671 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.740516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:36205 (local address 127.0.0.1:46508)
0923 14:40:28.740598 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.740633 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 14:40:28.740893 25267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.61:36205: Network error: Client connection negotiation failed: client connection to 127.2.218.61:36205: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab.
Entry for principal kudu/127.2.218.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab.
I20260923 14:40:28.749581 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.61:36205
--webserver_interface=127.2.218.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab
--principal=kudu/127.2.218.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:42327,127.2.218.61:36205,127.2.218.60:34205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:40:28.790199 25267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } attempt: 1
I20260923 14:40:28.791150 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.790867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:36205 (local address 127.0.0.1:46518)
0923 14:40:28.791023 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.791078 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 14:40:28.791801 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.791617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:36205 (local address 127.0.0.1:46534)
0923 14:40:28.791710 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.791754 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 14:40:28.791915 25267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.61:36205: Network error: Client connection negotiation failed: client connection to 127.2.218.61:36205: connect: Connection refused (error 111)
W20260923 14:40:28.844082 25274 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:40:28.844362 25274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:28.844434 25274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:28.844502 25274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:28.847800 25274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:28.847874 25274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:28.847908 25274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:28.847939 25274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:28.847970 25274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:28.852389 25274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:42327,127.2.218.61:36205,127.2.218.60:34205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab
--principal=kudu/127.2.218.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.61:36205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.218.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:28.853231 25274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:28.854086 25274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 14:40:28.860047 25267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } attempt: 2
I20260923 14:40:28.860823 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.860450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:36205 (local address 127.0.0.1:46542)
0923 14:40:28.860623 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.860686 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 14:40:28.861217 25279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:28.861359 25282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:28.861428 25274 server_base.cc:1034] running on GCE node
W20260923 14:40:28.861253 25280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:28.861671 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.861435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:36205 (local address 127.0.0.1:46544)
0923 14:40:28.861524 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.861573 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:36205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 14:40:28.861799 25267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.61:36205: Network error: Client connection negotiation failed: client connection to 127.2.218.61:36205: connect: Connection refused (error 111)
I20260923 14:40:28.862250 25274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:28.863106 25274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:28.864250 25274 hybrid_clock.cc:648] HybridClock initialized: now 1790174428864225 us; error 36 us; skew 500 ppm
Sep 23 14:40:28 dist-test-slave-zl47 krb5kdc[25186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:28.866570 25274 init.cc:377] Logged in from keytab as kudu/127.2.218.61@KRBTEST.COM (short username kudu)
I20260923 14:40:28.867216 25274 webserver.cc:460] Webserver started at http://127.2.218.61:39967/ using document root <none> and password file <none>
I20260923 14:40:28.867642 25274 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:28.867730 25274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:28.867928 25274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:28.869570 25274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/data/instance:
uuid: "dab9b22d21404fcc9fc7d9705740eb39"
format_stamp: "Formatted at 2026-09-23 14:40:28 on dist-test-slave-zl47"
server_key: "b6ec51796767705bdeb0fdd80dc05e8d"
server_key_iv: "d8f6fed8801af5f65e697fcc00356698"
server_key_version: "encryptionkey@0"
I20260923 14:40:28.870040 25274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/wal/instance:
uuid: "dab9b22d21404fcc9fc7d9705740eb39"
format_stamp: "Formatted at 2026-09-23 14:40:28 on dist-test-slave-zl47"
server_key: "b6ec51796767705bdeb0fdd80dc05e8d"
server_key_iv: "d8f6fed8801af5f65e697fcc00356698"
server_key_version: "encryptionkey@0"
I20260923 14:40:28.873044 25274 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:40:28.875098 25289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:28.876057 25274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 14:40:28.876176 25274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/wal
uuid: "dab9b22d21404fcc9fc7d9705740eb39"
format_stamp: "Formatted at 2026-09-23 14:40:28 on dist-test-slave-zl47"
server_key: "b6ec51796767705bdeb0fdd80dc05e8d"
server_key_iv: "d8f6fed8801af5f65e697fcc00356698"
server_key_version: "encryptionkey@0"
I20260923 14:40:28.876266 25274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:28.896987 25274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:28.899405 25274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:28.899574 25274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:28.905534 25274 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.61:36205
I20260923 14:40:28.905835 25341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.61:36205 every 8 connection(s)
I20260923 14:40:28.906441 25274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
I20260923 14:40:28.908896 25342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:28.912052 25342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:40:28.914320 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 25274
I20260923 14:40:28.914446 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-1/wal/instance
I20260923 14:40:28.914786 2920 external_mini_cluster.cc:1428] Setting key 9cc67b534d4d5a71f49ad7f227ea74a7
Sep 23 14:40:28 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.60@KRBTEST.COM" created.
I20260923 14:40:28.925247 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.913564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.2.218.62:42327)
0923 14:40:28.913704 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:28.913708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:28.914435 (+ 727us) server_negotiation.cc:407] Connection header received
0923 14:40:28.915535 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:28.915541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:28.915603 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:28.915691 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:28.916941 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.917386 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.918502 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.918730 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.918844 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:28.921258 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:28.921262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:28.921270 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:28.921295 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:28.922950 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:28.923542 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:28.923551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:28.923553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:28.923634 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:28.924069 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:28.924072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:28.924074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:28.924262 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:28.924374 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:28.924942 (+ 568us) server_negotiation.cc:299] Negotiation successful
0923 14:40:28.925079 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 14:40:28.926014 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.913869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:42327 (local address 127.0.0.1:46228)
0923 14:40:28.914293 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.914330 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:28.915248 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:28.915819 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:28.915825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:28.916154 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:28.916785 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:28.916804 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.917514 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.917517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:28.918380 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:28.918390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.918872 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.918876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:28.919033 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:28.919563 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:28.919577 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:28.921080 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:28.923093 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:28.923100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:28.923114 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:28.923396 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:28.923743 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:28.923746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:28.923748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:28.923928 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:28.924419 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:28.924428 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:28.924691 (+ 263us) client_negotiation.cc:769] Sending connection context
0923 14:40:28.924879 (+ 188us) client_negotiation.cc:240] Negotiation successful
0923 14:40:28.925178 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":91,"threads_started":1}
I20260923 14:40:28.927320 25342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:40:28 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
I20260923 14:40:28.936739 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.927792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:36205 (local address 127.0.0.1:46546)
0923 14:40:28.927899 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.927912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:28.927984 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:28.928488 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:28.928491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:28.928647 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:28.928711 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:28.928716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.929644 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.929649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:28.930177 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:28.930185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.930740 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.930743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:28.930873 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:28.931281 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:28.931297 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:28.933782 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:28.935528 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:28.935531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:28.935532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:28.935693 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:28.935934 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:28.935936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:28.935937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:28.935998 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:28.936387 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:28.936389 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:28.936461 (+ 72us) client_negotiation.cc:769] Sending connection context
0923 14:40:28.936558 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 14:40:28.936641 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":20}
I20260923 14:40:28.936914 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.927899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.2.218.61:36205)
0923 14:40:28.928195 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:28.928204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:28.928219 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:40:28.928248 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:28.928251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:28.928378 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:28.928508 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:28.928821 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.929530 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.930320 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.930634 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.930743 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:28.933911 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:28.933933 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:28.933941 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:28.933958 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:28.935372 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:28.935784 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:28.935788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:28.935789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:28.935860 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:28.936059 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:28.936097 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:28.936101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:28.936301 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:28.936392 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:28.936629 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 14:40:28.936731 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab.
Entry for principal kudu/127.2.218.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab.
I20260923 14:40:28.938473 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.60:34205
--webserver_interface=127.2.218.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab
--principal=kudu/127.2.218.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:40289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:42327,127.2.218.61:36205,127.2.218.60:34205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:40:28.938658 25342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:40:28.939231 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.939030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:41924)
0923 14:40:28.939120 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.939181 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 14:40:28.939432 25291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
I20260923 14:40:28.940208 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.940038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:41930)
0923 14:40:28.940128 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.940166 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 14:40:28.940328 25342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:34205: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
I20260923 14:40:28.952960 25267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } attempt: 3
Sep 23 14:40:28 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
I20260923 14:40:28.960814 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.953361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:36205 (local address 127.0.0.1:46562)
0923 14:40:28.953497 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.953511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:28.953621 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:28.953832 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:28.953834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:28.953988 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:28.954080 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:28.954084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.954755 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.954758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:28.955641 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:28.955650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.956103 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.956106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:28.956209 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:28.956611 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:28.956621 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:28.957881 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:28.959543 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:28.959546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:28.959548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:28.959726 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:28.959973 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:28.959976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:28.959977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:28.960054 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:28.960371 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:28.960373 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:28.960462 (+ 89us) client_negotiation.cc:769] Sending connection context
0923 14:40:28.960560 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 14:40:28.960682 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 14:40:28.960813 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.953410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.2.218.61:36205)
0923 14:40:28.953501 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:28.953507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:28.953571 (+ 64us) server_negotiation.cc:407] Connection header received
0923 14:40:28.953701 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:28.953704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:28.953744 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:28.953815 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:28.954217 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.954657 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.955758 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:28.955997 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:28.956092 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:28.958012 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:28.958017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:28.958023 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:28.958042 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:28.959434 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:28.959814 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:28.959818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:28.959820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:28.959889 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:28.960135 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:28.960138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:28.960139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:28.960282 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:28.960376 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:28.960604 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 14:40:28.960696 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 14:40:28.961833 25267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:40:28.962491 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.962276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:41944)
0923 14:40:28.962382 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.962432 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 14:40:28.963102 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.962904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:41952)
0923 14:40:28.963028 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.963064 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260923 14:40:28.963255 25267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:34205: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
I20260923 14:40:28.989540 25342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } attempt: 1
I20260923 14:40:28.990082 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.989884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:41966)
0923 14:40:28.989989 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.990032 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 14:40:28.990677 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.990514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:41968)
0923 14:40:28.990599 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.990636 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 14:40:28.990856 25342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:34205: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
I20260923 14:40:28.994423 25267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } attempt: 1
I20260923 14:40:28.994944 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.994732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:41984)
0923 14:40:28.994824 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.994882 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 14:40:28.995476 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:28.995317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:41988)
0923 14:40:28.995398 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:28.995436 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 14:40:28.995620 25267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:34205: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
W20260923 14:40:29.034516 25348 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:40:29.034708 25348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:29.034726 25348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:29.034806 25348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:29.038444 25348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:29.038486 25348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:29.038503 25348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:29.038517 25348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:29.038532 25348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:29.043326 25348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:40289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:42327,127.2.218.61:36205,127.2.218.60:34205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab
--principal=kudu/127.2.218.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.60:34205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.218.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:29.044315 25348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:29.045207 25348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:29.052784 25355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:29.052769 25357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:29.053047 25348 server_base.cc:1034] running on GCE node
W20260923 14:40:29.053180 25354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:29.053678 25348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:29.054672 25348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:29.055824 25348 hybrid_clock.cc:648] HybridClock initialized: now 1790174429055798 us; error 39 us; skew 500 ppm
Sep 23 14:40:29 dist-test-slave-zl47 krb5kdc[25186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:29.058380 25348 init.cc:377] Logged in from keytab as kudu/127.2.218.60@KRBTEST.COM (short username kudu)
I20260923 14:40:29.059016 25342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } attempt: 2
I20260923 14:40:29.059275 25348 webserver.cc:460] Webserver started at http://127.2.218.60:45593/ using document root <none> and password file <none>
I20260923 14:40:29.059710 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.059450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:41996)
0923 14:40:29.059580 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:29.059643 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 14:40:29.059885 25348 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:29.059968 25348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:29.060201 25348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:29.060442 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.060228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:41998)
0923 14:40:29.060335 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:29.060382 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 14:40:29.060647 25342 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:34205: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
I20260923 14:40:29.062167 25348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/data/instance:
uuid: "1103b9fabdd74590bacaa24f25e7fd79"
format_stamp: "Formatted at 2026-09-23 14:40:29 on dist-test-slave-zl47"
server_key: "f8683a35d603a1f1433ad86181df8a4b"
server_key_iv: "9a097576cb21a54d325dfd400fc811c4"
server_key_version: "encryptionkey@0"
I20260923 14:40:29.062673 25348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/wal/instance:
uuid: "1103b9fabdd74590bacaa24f25e7fd79"
format_stamp: "Formatted at 2026-09-23 14:40:29 on dist-test-slave-zl47"
server_key: "f8683a35d603a1f1433ad86181df8a4b"
server_key_iv: "9a097576cb21a54d325dfd400fc811c4"
server_key_version: "encryptionkey@0"
I20260923 14:40:29.065819 25348 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260923 14:40:29.068186 25364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:29.069284 25348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:29.069413 25348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/wal
uuid: "1103b9fabdd74590bacaa24f25e7fd79"
format_stamp: "Formatted at 2026-09-23 14:40:29 on dist-test-slave-zl47"
server_key: "f8683a35d603a1f1433ad86181df8a4b"
server_key_iv: "9a097576cb21a54d325dfd400fc811c4"
server_key_version: "encryptionkey@0"
I20260923 14:40:29.069506 25348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:29.070780 25267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } attempt: 2
I20260923 14:40:29.071414 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.071172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:42002)
0923 14:40:29.071287 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:29.071333 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 14:40:29.072041 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.071842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:42016)
0923 14:40:29.071949 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:29.071995 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 14:40:29.072185 25267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:34205: Network error: Client connection negotiation failed: client connection to 127.2.218.60:34205: connect: Connection refused (error 111)
I20260923 14:40:29.080331 25348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:29.082866 25348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:29.083048 25348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:29.088881 25348 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.60:34205
I20260923 14:40:29.089112 25416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.60:34205 every 8 connection(s)
I20260923 14:40:29.089646 25348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
I20260923 14:40:29.091849 25417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:29.093369 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 25348
I20260923 14:40:29.093520 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790174173335390-2920-0/minicluster-data/master-2/wal/instance
I20260923 14:40:29.093786 2920 external_mini_cluster.cc:1428] Setting key d242101ffc298bdb6910f24babf5a061
I20260923 14:40:29.095543 25417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:40:29 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
Sep 23 14:40:29 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:29.105367 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.097527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.2.218.62:42327)
0923 14:40:29.097635 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.097640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.097874 (+ 234us) server_negotiation.cc:407] Connection header received
0923 14:40:29.097990 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.097993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.098034 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.098125 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.098729 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.099133 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.099954 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.100127 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.100225 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.102503 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.102520 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.102525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.102548 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.103868 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.104221 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.104225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.104227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.104298 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.104562 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.104566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.104567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.104764 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.104858 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.105150 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.105244 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 14:40:29.106032 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.094946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.2.218.60:34205)
0923 14:40:29.095327 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.095343 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.095377 (+ 34us) server_negotiation.cc:407] Connection header received
0923 14:40:29.096036 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.096055 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.096406 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.096695 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.097279 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.098422 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.099279 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.099672 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.099772 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.102015 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.102020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.102036 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.102054 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.103778 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.104177 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.104181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.104182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.104269 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.104522 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.104525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.104527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.104795 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.104992 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.105174 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.105424 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":254,"thread_start_us":143,"threads_started":1}
I20260923 14:40:29.106034 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.097436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:42327 (local address 127.0.0.1:46236)
0923 14:40:29.097763 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:29.097802 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:29.097911 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:29.098190 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:29.098194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:29.098450 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:29.098613 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:29.098619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.099243 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.099246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:29.099846 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:29.099856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.100255 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.100258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:29.100355 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.100768 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:29.100780 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:29.102378 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:29.103934 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:29.103942 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:29.103949 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:29.104131 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:29.104382 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:29.104384 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:29.104386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:29.104481 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:29.104858 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:29.104864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:29.104978 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 14:40:29.105112 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 14:40:29.105643 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":324,"thread_start_us":119,"threads_started":1}
I20260923 14:40:29.107053 25417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:40:29 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
Sep 23 14:40:29 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:29.114537 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.107520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.2.218.61:36205)
0923 14:40:29.107645 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.107650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.107662 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:40:29.107703 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.107705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.107740 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.107816 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.108178 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.108844 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.109688 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.109966 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.110057 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.111670 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.111675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.111678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.111706 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.113044 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.113467 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.113469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.113470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.113536 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.113822 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.113825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.113826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.113992 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.114061 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.114363 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.114442 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 14:40:29.114557 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.107412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:36205 (local address 127.0.0.1:46578)
0923 14:40:29.107527 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:29.107536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:29.107617 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:29.107826 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:29.107828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:29.107973 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:29.108065 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:29.108086 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.108936 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.108939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:29.109592 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:29.109602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.110060 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.110063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:29.110162 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.110527 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:29.110538 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:29.111548 (+ 1010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:29.113139 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:29.113142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:29.113145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:29.113354 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:29.113624 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:29.113630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:29.113634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:29.113715 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:29.114086 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:29.114088 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:29.114191 (+ 103us) client_negotiation.cc:769] Sending connection context
0923 14:40:29.114298 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 14:40:29.114408 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 14:40:29.115195 25417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:40:29.116029 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.108049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.2.218.62:42327)
0923 14:40:29.108157 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.108162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.108190 (+ 28us) server_negotiation.cc:407] Connection header received
0923 14:40:29.108285 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.108287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.108313 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.108357 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.108934 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.109558 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.110460 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.110733 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.110832 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.112982 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.112986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.112988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.113018 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.114454 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.114884 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.114887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.114888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.114965 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.115286 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.115288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.115290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.115483 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.115692 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.115807 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.115906 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 23 14:40:29 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
Sep 23 14:40:29 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
I20260923 14:40:29.122174 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.115534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:42022)
0923 14:40:29.115681 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:29.115692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:29.115765 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:29.115971 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:29.115974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:29.116111 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:29.116187 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:29.116192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.116846 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.116849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:29.117564 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:29.117574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.117952 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.117955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:29.118067 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.118462 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:29.118474 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:29.119590 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:29.120966 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:29.120968 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:29.120969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:29.121135 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:29.121365 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:29.121370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:29.121371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:29.121442 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:29.121743 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:29.121746 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:29.121809 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 14:40:29.121916 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 14:40:29.122022 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":22}
I20260923 14:40:29.122195 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.115605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.2.218.60:34205)
0923 14:40:29.115688 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.115692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.115731 (+ 39us) server_negotiation.cc:407] Connection header received
0923 14:40:29.115826 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.115828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.115861 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.115940 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.116269 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.116734 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.117665 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.117847 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.117954 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.119704 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.119706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.119707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.119723 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.120865 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.121217 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.121220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.121222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.121297 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.121522 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.121524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.121526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.121659 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.121743 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.121953 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.122065 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":33}
I20260923 14:40:29.124475 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.116934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.2.218.61:36205)
0923 14:40:29.117015 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.117018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.117036 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:40:29.117108 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.117110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.117136 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.117185 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.117753 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.118225 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.119112 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.119434 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.119501 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.121694 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.121697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.121699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.121718 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.123182 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.123527 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.123529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.123531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.123599 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.123814 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.123817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.123818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.124008 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.124206 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.124269 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.124359 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:40:29.125056 25417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79: Bootstrap starting.
I20260923 14:40:29.127012 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:29.127303 25417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:29.128049 25417 log.cc:824] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:29.128286 25241 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f05bde479864490f8cf2646d47016741"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46246
I20260923 14:40:29.129755 25417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79: No bootstrap required, opened a new log
I20260923 14:40:29.133045 25417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } }
I20260923 14:40:29.133239 25417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1103b9fabdd74590bacaa24f25e7fd79, State: Initialized, Role: FOLLOWER
I20260923 14:40:29.133512 25417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } }
I20260923 14:40:29.133870 25423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } } }
I20260923 14:40:29.133985 25423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:29.134539 25417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 14:40:29.138473 25434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:40:29.138542 25434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:40:29.137388 25348 master_runner.cc:186] Error getting master registration for 127.2.218.62:42327: OK, instance_id { permanent_uuid: "f05bde479864490f8cf2646d47016741" instance_seqno: 1790174428715770 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:40:29.140686 25348 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:40:29.151796 25342 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } attempt: 3
Sep 23 14:40:29 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
I20260923 14:40:29.159212 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.152192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.2.218.60:34205)
0923 14:40:29.152275 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.152278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.152290 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:40:29.152383 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.152386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.152418 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.152485 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.152856 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.153481 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.154420 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.154698 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.154784 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.156622 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.156625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.156627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.156652 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.157829 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.158218 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.158221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.158222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.158288 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.158527 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.158531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.158533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.158689 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.158783 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.159012 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.159096 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 14:40:29.159212 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.152148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:42026)
0923 14:40:29.152238 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:29.152250 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:29.152314 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:29.152517 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:29.152519 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:29.152670 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:29.152754 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:29.152760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.153572 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.153574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:29.154312 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:29.154320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.154793 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.154797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:29.154879 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.155272 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:29.155283 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:29.156502 (+ 1219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:29.157933 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:29.157937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:29.157939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:29.158143 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:29.158367 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:29.158370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:29.158372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:29.158449 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:29.158778 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:29.158780 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:29.158867 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 14:40:29.158970 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 14:40:29.159085 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 14:40:29.161542 25342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39: Bootstrap starting.
I20260923 14:40:29.163928 25342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:29.164703 25342 log.cc:824] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:29.166503 25342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39: No bootstrap required, opened a new log
I20260923 14:40:29.170055 25342 raft_consensus.cc:348] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } }
I20260923 14:40:29.170254 25342 raft_consensus.cc:729] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab9b22d21404fcc9fc7d9705740eb39, State: Initialized, Role: FOLLOWER
I20260923 14:40:29.170508 25342 consensus_queue.cc:260] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } }
I20260923 14:40:29.170893 25437 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } } }
I20260923 14:40:29.171003 25437 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:29.171286 25267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } attempt: 3
I20260923 14:40:29.171634 25342 sys_catalog.cc:564] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [sys.catalog]: configured and running, proceeding with master startup.
Sep 23 14:40:29 dist-test-slave-zl47 krb5kdc[25186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174428, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
W20260923 14:40:29.177003 25448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:40:29.177107 25448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:40:29.174750 25274 master_runner.cc:186] Error getting master registration for 127.2.218.62:42327: OK, instance_id { permanent_uuid: "f05bde479864490f8cf2646d47016741" instance_seqno: 1790174428715770 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:40:29.178737 25274 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:40:29.180047 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.171633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:34205 (local address 127.0.0.1:42034)
0923 14:40:29.171897 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:29.171908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:29.171976 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:29.172201 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:29.172204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:29.172357 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:29.172439 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:29.172444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.173254 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.173258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:29.174162 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:29.174173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.174851 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.174854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:29.174946 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.175354 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:29.175383 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:29.176696 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:29.178567 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:29.178570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:29.178572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:29.178815 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:29.179135 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:29.179138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:29.179140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:29.179233 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:29.179631 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:29.179633 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:29.179713 (+ 80us) client_negotiation.cc:769] Sending connection context
0923 14:40:29.179800 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 14:40:29.179913 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260923 14:40:29.180480 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.171790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.2.218.60:34205)
0923 14:40:29.171882 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.171887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.171947 (+ 60us) server_negotiation.cc:407] Connection header received
0923 14:40:29.172065 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.172068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.172106 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.172169 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.172534 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.173147 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.174334 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.174656 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.174751 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.176843 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.176846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.176849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.176869 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.178469 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.178934 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.178950 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.178953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.179030 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.179312 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.179315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.179316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.179539 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.179628 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.180248 (+ 620us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.180351 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 14:40:29.183043 25267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741: Bootstrap starting.
I20260923 14:40:29.185307 25267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:29.186054 25267 log.cc:824] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:29.187917 25267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741: No bootstrap required, opened a new log
I20260923 14:40:29.190544 25267 raft_consensus.cc:348] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } }
I20260923 14:40:29.190737 25267 raft_consensus.cc:729] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f05bde479864490f8cf2646d47016741, State: Initialized, Role: FOLLOWER
I20260923 14:40:29.191010 25267 consensus_queue.cc:260] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } }
I20260923 14:40:29.192055 25267 sys_catalog.cc:564] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:29.192113 25241 raft_consensus.cc:482] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 14:40:29.192211 25241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:29.193199 25241 raft_consensus.cc:504] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } }
I20260923 14:40:29.193874 25451 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } } }
I20260923 14:40:29.193972 25451 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:29.193984 25241 leader_election.cc:288] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [CANDIDATE]: Term 1 election: Requested vote from peers dab9b22d21404fcc9fc7d9705740eb39 (127.2.218.61:36205), 1103b9fabdd74590bacaa24f25e7fd79 (127.2.218.60:34205)
I20260923 14:40:29.194667 25316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f05bde479864490f8cf2646d47016741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dab9b22d21404fcc9fc7d9705740eb39"
I20260923 14:40:29.194805 25391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f05bde479864490f8cf2646d47016741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1103b9fabdd74590bacaa24f25e7fd79"
I20260923 14:40:29.194890 25316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:29.195006 25391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:29.196022 25316 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f05bde479864490f8cf2646d47016741 in term 1.
I20260923 14:40:29.196096 25391 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f05bde479864490f8cf2646d47016741 in term 1.
I20260923 14:40:29.196517 25216 leader_election.cc:302] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1103b9fabdd74590bacaa24f25e7fd79, f05bde479864490f8cf2646d47016741; no voters:
W20260923 14:40:29.198127 25462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:40:29.198236 25462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:40:29.198268 25451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:29.198521 25451 raft_consensus.cc:686] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [term 1 LEADER]: Becoming Leader. State: Replica: f05bde479864490f8cf2646d47016741, State: Running, Role: LEADER
I20260923 14:40:29.198809 25451 consensus_queue.cc:237] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } }
I20260923 14:40:29.200829 25468 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f05bde479864490f8cf2646d47016741. Latest consensus state: current_term: 1 leader_uuid: "f05bde479864490f8cf2646d47016741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } } }
I20260923 14:40:29.200932 25468 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:29.202023 25199 master_runner.cc:263] No leader master found from master f05bde479864490f8cf2646d47016741
I20260923 14:40:29.202071 25199 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:40:29.203074 25470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:29.203841 25470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:29.210518 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.201791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46256 (local address 127.2.218.62:42327)
0923 14:40:29.201960 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.201964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.202504 (+ 540us) server_negotiation.cc:407] Connection header received
0923 14:40:29.202654 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.202657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.202692 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.202732 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.203631 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.204295 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.205452 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.205773 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.205849 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.207469 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.207473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.207476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.207497 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.208905 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.209317 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.209320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.209322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.209391 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.209660 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.209662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.209664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.209842 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.209955 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.210288 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.210384 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260923 14:40:29.212707 25220 tablet.cc:2369] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:40:29.217393 25391 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [term 1 FOLLOWER]: Refusing update from remote peer f05bde479864490f8cf2646d47016741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:29.218039 25468 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:29.218176 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.201429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.2.218.61:36205)
0923 14:40:29.201532 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.201537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.205864 (+ 4327us) server_negotiation.cc:407] Connection header received
0923 14:40:29.206007 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.206010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.206047 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.206087 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.206954 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.207562 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.213110 (+ 5548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.213416 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.213479 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.214957 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.214960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.214962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.214984 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.216504 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.216935 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.216938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.216940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.217015 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.217277 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.217280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.217281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.217476 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.217570 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.217895 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.218006 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 14:40:29.219597 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:29.202546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.2.218.60:34205)
0923 14:40:29.202640 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.202644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.211824 (+ 9180us) server_negotiation.cc:407] Connection header received
0923 14:40:29.211972 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.211975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.212013 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.212054 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.212752 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.213390 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.214379 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:29.214661 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:29.214721 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:29.216160 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:29.216163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:29.216165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:29.216187 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:29.217589 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.218162 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.218165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.218167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.218247 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:29.218760 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:29.218764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:29.218765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:29.218975 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:29.219076 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:29.219374 (+ 298us) server_negotiation.cc:299] Negotiation successful
0923 14:40:29.219472 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 14:40:29.222985 25423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f05bde479864490f8cf2646d47016741. Latest consensus state: current_term: 1 leader_uuid: "f05bde479864490f8cf2646d47016741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } } }
I20260923 14:40:29.223089 25423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:29.223990 25316 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [term 1 FOLLOWER]: Refusing update from remote peer f05bde479864490f8cf2646d47016741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:29.225131 25451 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:29.226668 25470 catalog_manager.cc:1347] Generated new cluster ID: 883e0d8a30274f709f5cf1179aae847b
I20260923 14:40:29.226727 25470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:29.229172 25423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f05bde479864490f8cf2646d47016741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } } }
I20260923 14:40:29.229280 25423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:29.229687 25468 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f05bde479864490f8cf2646d47016741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } } }
I20260923 14:40:29.229772 25468 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:29.229880 25474 mvcc.cc:205] Tried to move back new op lower bound from 7332554462067453952 to 7332554462001651712. Current Snapshot: MvccSnapshot[applied={T|T < 7332554462067453952 or (T in {7332554462067453952})}]
I20260923 14:40:29.234382 25481 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f05bde479864490f8cf2646d47016741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } } }
I20260923 14:40:29.234583 25481 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:29.235287 25437 sys_catalog.cc:455] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f05bde479864490f8cf2646d47016741. Latest consensus state: current_term: 1 leader_uuid: "f05bde479864490f8cf2646d47016741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } } }
I20260923 14:40:29.235420 25437 sys_catalog.cc:458] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:29.236627 25451 sys_catalog.cc:455] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f05bde479864490f8cf2646d47016741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f05bde479864490f8cf2646d47016741" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 42327 } } peers { permanent_uuid: "dab9b22d21404fcc9fc7d9705740eb39" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 36205 } } peers { permanent_uuid: "1103b9fabdd74590bacaa24f25e7fd79" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 34205 } } }
I20260923 14:40:29.236721 25451 sys_catalog.cc:458] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:29.242143 25470 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:29.243266 25470 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:29.247888 25479 mvcc.cc:205] Tried to move back new op lower bound from 7332554462067453952 to 7332554462001651712. Current Snapshot: MvccSnapshot[applied={T|T < 7332554462067453952}]
I20260923 14:40:29.250185 25475 mvcc.cc:205] Tried to move back new op lower bound from 7332554462067453952 to 7332554462001651712. Current Snapshot: MvccSnapshot[applied={T|T < 7332554462067453952}]
I20260923 14:40:29.256346 25470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f05bde479864490f8cf2646d47016741: Generated new TSK 0
I20260923 14:40:29.257067 25470 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:40:29.281311 25269 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.278617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.2.218.62:42327)
0923 14:40:29.278748 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.278753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.280278 (+ 1525us) server_negotiation.cc:407] Connection header received
0923 14:40:29.280475 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.280479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.280526 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.280578 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.281192 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46264: BlockingRecv error: recv got EOF from 127.0.0.1:46264 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260923 14:40:29.282990 25420 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.279299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.2.218.60:34205)
0923 14:40:29.279388 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.279392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.281654 (+ 2262us) server_negotiation.cc:407] Connection header received
0923 14:40:29.281953 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.281956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.281998 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.282044 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.282869 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42064: BlockingRecv error: recv got EOF from 127.0.0.1:42064 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260923 14:40:29.283871 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.279904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.2.218.61:36205)
0923 14:40:29.280013 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.280017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.282295 (+ 2278us) server_negotiation.cc:407] Connection header received
0923 14:40:29.282407 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.282410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.282455 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.282542 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.283750 (+ 1208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46610: BlockingRecv error: recv got EOF from 127.0.0.1:46610 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 14:40:29.287192 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.285987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.2.218.61:36205)
0923 14:40:29.286081 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.286085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.286374 (+ 289us) server_negotiation.cc:407] Connection header received
0923 14:40:29.286568 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.286571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.286636 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.286678 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.287110 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46618: BlockingRecv error: recv got EOF from 127.0.0.1:46618 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260923 14:40:29.291610 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.290297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.2.218.61:36205)
0923 14:40:29.290381 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.290385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.290811 (+ 426us) server_negotiation.cc:407] Connection header received
0923 14:40:29.291016 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.291020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.291067 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.291119 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.291535 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46620: BlockingRecv error: recv got EOF from 127.0.0.1:46620 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260923 14:40:29.296518 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.295493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.2.218.61:36205)
0923 14:40:29.295591 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.295594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.295842 (+ 248us) server_negotiation.cc:407] Connection header received
0923 14:40:29.295991 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.295994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.296028 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.296066 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.296448 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46630: BlockingRecv error: recv got EOF from 127.0.0.1:46630 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 14:40:29.304535 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.303370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.2.218.61:36205)
0923 14:40:29.303475 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.303479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.303854 (+ 375us) server_negotiation.cc:407] Connection header received
0923 14:40:29.304016 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.304019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.304054 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.304093 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.304468 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46638: BlockingRecv error: recv got EOF from 127.0.0.1:46638 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260923 14:40:29.315476 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.314243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.2.218.61:36205)
0923 14:40:29.314318 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.314321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.314701 (+ 380us) server_negotiation.cc:407] Connection header received
0923 14:40:29.314849 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.314852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.314901 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.314960 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.315368 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46648: BlockingRecv error: recv got EOF from 127.0.0.1:46648 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260923 14:40:29.334928 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.333386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.2.218.61:36205)
0923 14:40:29.333490 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.333495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.333636 (+ 141us) server_negotiation.cc:407] Connection header received
0923 14:40:29.333735 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.333738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.333781 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.333862 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.334846 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46662: BlockingRecv error: recv got EOF from 127.0.0.1:46662 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 14:40:29.369561 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.368201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.2.218.61:36205)
0923 14:40:29.368314 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.368319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.368535 (+ 216us) server_negotiation.cc:407] Connection header received
0923 14:40:29.368853 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.368857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.368921 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.369017 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.369465 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46672: BlockingRecv error: recv got EOF from 127.0.0.1:46672 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260923 14:40:29.436617 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.435467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.2.218.61:36205)
0923 14:40:29.435580 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.435585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.435788 (+ 203us) server_negotiation.cc:407] Connection header received
0923 14:40:29.435932 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.435935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.435985 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.436061 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.436519 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46684: BlockingRecv error: recv got EOF from 127.0.0.1:46684 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260923 14:40:29.567634 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.566562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.2.218.61:36205)
0923 14:40:29.566700 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.566705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.566845 (+ 140us) server_negotiation.cc:407] Connection header received
0923 14:40:29.566963 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.566965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.567012 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.567084 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.567480 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46686: BlockingRecv error: recv got EOF from 127.0.0.1:46686 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260923 14:40:29.826541 25347 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:29.825490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.2.218.61:36205)
0923 14:40:29.825610 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:29.825613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:29.825721 (+ 108us) server_negotiation.cc:407] Connection header received
0923 14:40:29.825833 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:29.825835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:29.825875 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:29.825952 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:29.826411 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46688: BlockingRecv error: recv got EOF from 127.0.0.1:46688 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 14:40:30.139599 25434 catalog_manager.cc:1259] Loaded cluster ID: 883e0d8a30274f709f5cf1179aae847b
I20260923 14:40:30.139681 25434 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79: loading cluster ID for follower catalog manager: success
I20260923 14:40:30.141135 25434 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79: acquiring CA information for follower catalog manager: success
I20260923 14:40:30.142155 25434 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1103b9fabdd74590bacaa24f25e7fd79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:40:30.178073 25448 catalog_manager.cc:1259] Loaded cluster ID: 883e0d8a30274f709f5cf1179aae847b
I20260923 14:40:30.178176 25448 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39: loading cluster ID for follower catalog manager: success
I20260923 14:40:30.179610 25448 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39: acquiring CA information for follower catalog manager: success
I20260923 14:40:30.180617 25448 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dab9b22d21404fcc9fc7d9705740eb39: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:40:30.344334 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:30.340605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.2.218.61:36205)
0923 14:40:30.340849 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:30.340853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:30.340871 (+ 18us) server_negotiation.cc:407] Connection header received
0923 14:40:30.340983 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:30.340985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:30.341086 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:30.341191 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:30.341423 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:30.341955 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:30.343053 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:30.343253 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:30.343334 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:30.343584 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:30.343967 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:30.344053 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:30.344103 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 14:40:30.344147 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":64,"threads_started":1}
I20260923 14:40:30.349746 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:30.346923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42070 (local address 127.2.218.60:34205)
0923 14:40:30.347099 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:30.347102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:30.347113 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:40:30.347207 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:30.347209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:30.347297 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:30.347370 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:30.347556 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:30.347983 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:30.348648 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:30.348830 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:30.348902 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:30.349102 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:30.349411 (+ 309us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:30.349486 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:30.349552 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 14:40:30.349591 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260923 14:40:30.355235 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:30.352067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.2.218.61:36205)
0923 14:40:30.352156 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:30.352159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:30.352414 (+ 255us) server_negotiation.cc:407] Connection header received
0923 14:40:30.352525 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:30.352528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:30.352577 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:30.352645 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:30.352861 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:30.353456 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:30.354216 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:30.354487 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:30.354590 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:30.354817 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:30.354888 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:30.354957 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:30.355072 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 14:40:30.355129 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:40:30.355834 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:30.352464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.2.218.60:34205)
0923 14:40:30.352568 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:30.352572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:30.352891 (+ 319us) server_negotiation.cc:407] Connection header received
0923 14:40:30.352977 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:30.352980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:30.353015 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:30.353089 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:30.353280 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:30.353840 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:30.354755 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:30.355037 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:30.355099 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:30.355430 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:30.355491 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:30.355526 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:30.355673 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 14:40:30.355713 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 14:40:30.356575 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:30.352146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.2.218.62:42327)
0923 14:40:30.352355 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:30.352365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:30.352827 (+ 462us) server_negotiation.cc:407] Connection header received
0923 14:40:30.352922 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:30.352925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:30.353057 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:30.353161 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:30.353515 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:30.354197 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:30.355161 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:30.355502 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:30.355608 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:30.355874 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:30.356128 (+ 254us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:30.356216 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:30.356351 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 14:40:30.356424 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260923 14:40:30.359624 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 25199
I20260923 14:40:30.365918 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 25274
I20260923 14:40:30.370913 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 25348
2026-09-23T14:40:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2885 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 14:40:30 dist-test-slave-zl47 krb5kdc[25568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 14:40:30 dist-test-slave-zl47 krb5kdc[25568](info): set up 2 sockets
Sep 23 14:40:30 dist-test-slave-zl47 krb5kdc[25568](info): commencing operation
krb5kdc: starting...
W20260923 14:40:31.274924 2920 mini_kdc.cc:121] Time spent starting KDC: real 0.884s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/test-user.keytab.
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T14:40:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T14:40:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.218.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
Entry for principal kudu/127.2.218.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab.
I20260923 14:40:31.381042 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:46435
--webserver_interface=127.2.218.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:46435,127.2.218.61:34133,127.2.218.60:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 14:40:31.472357 25581 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:40:31.472637 25581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:31.472681 25581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:31.472740 25581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:31.476114 25581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:31.476186 25581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:31.476218 25581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:31.476249 25581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:31.476277 25581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:31.480712 25581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:46435,127.2.218.61:34133,127.2.218.60:41957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.62.keytab
--principal=kudu/127.2.218.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.62:46435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.218.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:31.481490 25581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:31.482259 25581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:31.489650 25589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:31.489741 25581 server_base.cc:1034] running on GCE node
W20260923 14:40:31.489658 25586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:31.489658 25587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:31.490350 25581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:31.491175 25581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:31.492324 25581 hybrid_clock.cc:648] HybridClock initialized: now 1790174431492309 us; error 25 us; skew 500 ppm
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:31.494598 25581 init.cc:377] Logged in from keytab as kudu/127.2.218.62@KRBTEST.COM (short username kudu)
I20260923 14:40:31.495464 25581 webserver.cc:460] Webserver started at http://127.2.218.62:35843/ using document root <none> and password file <none>
I20260923 14:40:31.495860 25581 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:31.495910 25581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:31.496037 25581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:31.497663 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/data/instance:
uuid: "88f61bb033b94f289e4e9aa80564e0b1"
format_stamp: "Formatted at 2026-09-23 14:40:31 on dist-test-slave-zl47"
server_key: "3e7b7e1af7399c5f0f844c4c0c015f1a"
server_key_iv: "a00847dd7156b762d019de0f656fc897"
server_key_version: "encryptionkey@0"
I20260923 14:40:31.498139 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/wal/instance:
uuid: "88f61bb033b94f289e4e9aa80564e0b1"
format_stamp: "Formatted at 2026-09-23 14:40:31 on dist-test-slave-zl47"
server_key: "3e7b7e1af7399c5f0f844c4c0c015f1a"
server_key_iv: "a00847dd7156b762d019de0f656fc897"
server_key_version: "encryptionkey@0"
I20260923 14:40:31.500972 25581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260923 14:40:31.502899 25596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:31.503825 25581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:31.503916 25581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/wal
uuid: "88f61bb033b94f289e4e9aa80564e0b1"
format_stamp: "Formatted at 2026-09-23 14:40:31 on dist-test-slave-zl47"
server_key: "3e7b7e1af7399c5f0f844c4c0c015f1a"
server_key_iv: "a00847dd7156b762d019de0f656fc897"
server_key_version: "encryptionkey@0"
I20260923 14:40:31.503981 25581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:31.515369 25581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:31.517387 25581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:31.517501 25581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:31.523185 25581 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.62:46435
I20260923 14:40:31.523412 25648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.62:46435 every 8 connection(s)
I20260923 14:40:31.523942 25581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/data/info.pb
I20260923 14:40:31.524998 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 25581
I20260923 14:40:31.525117 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-0/wal/instance
I20260923 14:40:31.525347 2920 external_mini_cluster.cc:1428] Setting key 14515430dd13b67525ae6666262b7530
I20260923 14:40:31.526262 25649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:31.530366 25649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.218.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.61@KRBTEST.COM" created.
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:31.542757 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.532688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.2.218.62:46435)
0923 14:40:31.532944 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.532952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.532985 (+ 33us) server_negotiation.cc:407] Connection header received
0923 14:40:31.533441 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.533445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.533597 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.533727 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.534398 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.535383 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.536447 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.536748 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.536903 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.538781 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.538785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.538792 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.538809 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.540275 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.540668 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.540672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.540674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.540761 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.541029 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.541032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.541033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.541318 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.541442 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.541747 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.541972 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":178,"thread_start_us":70,"threads_started":1}
I20260923 14:40:31.542757 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.532012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:46435 (local address 127.0.0.1:59454)
0923 14:40:31.532366 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.532415 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:31.533140 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:31.533750 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:31.533753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:31.534001 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:31.534271 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.534284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.535500 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.535506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.536331 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.536339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.536869 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.536872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.536950 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.537424 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:31.537440 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:31.538651 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:31.540368 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.540377 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.540393 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.540574 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.540847 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.540850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.540851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.540945 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.541424 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:31.541428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:31.541544 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 14:40:31.541676 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 14:40:31.542021 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":125,"thread_start_us":108,"threads_started":1}
I20260923 14:40:31.544548 25649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:40:31.545066 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.544864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34133 (local address 127.0.0.1:47486)
0923 14:40:31.544953 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.545013 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 14:40:31.545238 25599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34133: connect: Connection refused (error 111)
I20260923 14:40:31.545854 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.545680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34133 (local address 127.0.0.1:47494)
0923 14:40:31.545768 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.545805 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 14:40:31.545967 25649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.61:34133: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab.
Entry for principal kudu/127.2.218.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab.
I20260923 14:40:31.546744 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.61:34133
--webserver_interface=127.2.218.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab
--principal=kudu/127.2.218.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:46435,127.2.218.61:34133,127.2.218.60:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:40:31.595216 25649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } attempt: 1
I20260923 14:40:31.595803 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.595590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34133 (local address 127.0.0.1:47496)
0923 14:40:31.595701 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.595750 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 14:40:31.596411 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.596219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34133 (local address 127.0.0.1:47498)
0923 14:40:31.596308 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.596347 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 14:40:31.596563 25649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.61:34133: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34133: connect: Connection refused (error 111)
W20260923 14:40:31.636212 25656 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:40:31.636401 25656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:31.636420 25656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:31.636451 25656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:31.639734 25656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:31.639773 25656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:31.639789 25656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:31.639837 25656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:31.639860 25656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:31.644094 25656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:46435,127.2.218.61:34133,127.2.218.60:41957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.61.keytab
--principal=kudu/127.2.218.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.61:34133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.218.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:31.644810 25656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:31.645557 25656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:31.652444 25661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:31.652501 25662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:31.652594 25656 server_base.cc:1034] running on GCE node
W20260923 14:40:31.652519 25664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:31.653157 25656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:31.653976 25656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:31.655148 25656 hybrid_clock.cc:648] HybridClock initialized: now 1790174431655142 us; error 23 us; skew 500 ppm
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:31.657351 25656 init.cc:377] Logged in from keytab as kudu/127.2.218.61@KRBTEST.COM (short username kudu)
I20260923 14:40:31.658010 25656 webserver.cc:460] Webserver started at http://127.2.218.61:35813/ using document root <none> and password file <none>
I20260923 14:40:31.658501 25656 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:31.658578 25656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:31.658759 25656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:31.660280 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/data/instance:
uuid: "4f228f0a110548be9bcfa94207ce89c9"
format_stamp: "Formatted at 2026-09-23 14:40:31 on dist-test-slave-zl47"
server_key: "ade262f29b8aadd1eab5fa55881197bf"
server_key_iv: "fe665c90d83db3533845201e086db760"
server_key_version: "encryptionkey@0"
I20260923 14:40:31.660741 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/wal/instance:
uuid: "4f228f0a110548be9bcfa94207ce89c9"
format_stamp: "Formatted at 2026-09-23 14:40:31 on dist-test-slave-zl47"
server_key: "ade262f29b8aadd1eab5fa55881197bf"
server_key_iv: "fe665c90d83db3533845201e086db760"
server_key_version: "encryptionkey@0"
I20260923 14:40:31.663602 25656 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260923 14:40:31.664716 25649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } attempt: 2
I20260923 14:40:31.665318 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.665107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34133 (local address 127.0.0.1:47502)
0923 14:40:31.665216 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.665263 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 14:40:31.665583 25671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:31.665977 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.665790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34133 (local address 127.0.0.1:47504)
0923 14:40:31.665882 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.665931 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 14:40:31.666117 25649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.61:34133: Network error: Client connection negotiation failed: client connection to 127.2.218.61:34133: connect: Connection refused (error 111)
I20260923 14:40:31.666718 25656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:31.666848 25656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/wal
uuid: "4f228f0a110548be9bcfa94207ce89c9"
format_stamp: "Formatted at 2026-09-23 14:40:31 on dist-test-slave-zl47"
server_key: "ade262f29b8aadd1eab5fa55881197bf"
server_key_iv: "fe665c90d83db3533845201e086db760"
server_key_version: "encryptionkey@0"
I20260923 14:40:31.666956 25656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:31.692826 25656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:31.695603 25656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:31.695786 25656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:31.701951 25656 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.61:34133
I20260923 14:40:31.702217 25723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.61:34133 every 8 connection(s)
I20260923 14:40:31.702822 25656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/data/info.pb
I20260923 14:40:31.705029 25724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:31.707885 25724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:40:31.711189 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 25656
I20260923 14:40:31.711315 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-1/wal/instance
I20260923 14:40:31.711555 2920 external_mini_cluster.cc:1428] Setting key 87c848d8b1a087fbc09fd07fa23bbd95
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.218.60@KRBTEST.COM; defaulting to no policy
I20260923 14:40:31.719414 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.709254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.2.218.62:46435)
0923 14:40:31.709383 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.709386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.710030 (+ 644us) server_negotiation.cc:407] Connection header received
0923 14:40:31.710875 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.710878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.710931 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.711035 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.712106 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.712572 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.713579 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.713783 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.713894 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.716043 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.716052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.716056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.716076 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.717402 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.717952 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.717955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.717958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.718041 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.718412 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.718415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.718417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.718582 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.718680 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.719142 (+ 462us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.719253 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.218.60@KRBTEST.COM" created.
I20260923 14:40:31.720026 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.709524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:46435 (local address 127.0.0.1:59466)
0923 14:40:31.709902 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.709938 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:31.710691 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:31.711174 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:31.711179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:31.711548 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:31.711974 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.711986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.712696 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.712704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.713429 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.713438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.713923 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.713927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.714073 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.714631 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:31.714646 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:31.715892 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:31.717532 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.717537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.717548 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.717842 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.718168 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.718171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.718173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.718303 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.718690 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:31.718697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:31.718908 (+ 211us) client_negotiation.cc:769] Sending connection context
0923 14:40:31.719069 (+ 161us) client_negotiation.cc:240] Negotiation successful
0923 14:40:31.719329 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
I20260923 14:40:31.721155 25724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
I20260923 14:40:31.730062 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.721613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34133 (local address 127.0.0.1:47506)
0923 14:40:31.721757 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.721768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:31.721831 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:31.722316 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:31.722322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:31.722483 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:31.722598 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.722606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.723438 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.723441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.724304 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.724317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.724859 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.724863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.724960 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.725311 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:31.725328 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:31.726593 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:31.728686 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.728690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.728692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.728872 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.729182 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.729184 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.729186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.729257 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.729635 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:31.729638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:31.729706 (+ 68us) client_negotiation.cc:769] Sending connection context
0923 14:40:31.729805 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 14:40:31.729919 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 14:40:31.730134 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.721697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.2.218.61:34133)
0923 14:40:31.721954 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.721965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.721991 (+ 26us) server_negotiation.cc:407] Connection header received
0923 14:40:31.722024 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.722027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.722199 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.722296 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.722713 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.723307 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.724429 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.724747 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.724827 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.726729 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.726740 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.726751 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.726790 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.728554 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.728999 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.729004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.729005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.729078 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.729336 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.729339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.729341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.729534 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.729654 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.729883 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.729986 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":108,"threads_started":1}
I20260923 14:40:31.731547 25724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:40:31.732131 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.731916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:59928)
0923 14:40:31.732019 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.732085 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 14:40:31.732303 25675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
I20260923 14:40:31.733011 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.732821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:59934)
0923 14:40:31.732921 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.732960 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 14:40:31.733153 25724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:41957: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.218.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab.
Entry for principal kudu/127.2.218.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab.
I20260923 14:40:31.734620 2920 external_mini_cluster.cc:1326] Running /tmp/dist-test-task3TRGqf/build/debug/bin/kudu
/tmp/dist-test-task3TRGqf/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.60:41957
--webserver_interface=127.2.218.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab
--principal=kudu/127.2.218.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.218.20:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.218.62:46435,127.2.218.61:34133,127.2.218.60:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 14:40:31.757279 25649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } attempt: 3
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
I20260923 14:40:31.764874 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.757736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.2.218.61:34133)
0923 14:40:31.757827 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.757833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.757845 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:40:31.757957 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.757960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.758001 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.758083 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.758510 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.758955 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.760090 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.760349 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.760463 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.762262 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.762265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.762267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.762288 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.763496 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.763869 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.763872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.763873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.763947 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.764192 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.764196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.764197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.764339 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.764435 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.764665 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.764754 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:40:31.764874 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.757677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34133 (local address 127.0.0.1:47514)
0923 14:40:31.757785 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.757798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:31.757884 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:31.758128 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:31.758130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:31.758300 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:31.758387 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.758393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.759055 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.759058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.759966 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.759975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.760467 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.760470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.760585 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.761069 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:31.761079 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:31.762146 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:31.763607 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.763609 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.763611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.763781 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.764031 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.764033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.764035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.764112 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.764435 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:31.764439 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:31.764526 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 14:40:31.764621 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 14:40:31.764739 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 14:40:31.766182 25649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } has no permanent_uuid. Determining permanent_uuid...
I20260923 14:40:31.766754 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.766564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:59946)
0923 14:40:31.766657 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.766706 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 14:40:31.767333 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.767129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:59948)
0923 14:40:31.767244 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.767283 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260923 14:40:31.767462 25649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:41957: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
I20260923 14:40:31.782321 25724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } attempt: 1
I20260923 14:40:31.782801 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.782627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:59962)
0923 14:40:31.782716 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.782755 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 14:40:31.783375 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.783198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:59966)
0923 14:40:31.783288 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.783332 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 14:40:31.783496 25724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:41957: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
I20260923 14:40:31.798596 25649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } attempt: 1
I20260923 14:40:31.799089 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.798918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:59972)
0923 14:40:31.799009 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.799048 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 14:40:31.799620 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.799454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:59988)
0923 14:40:31.799540 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.799581 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 14:40:31.799724 25649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:41957: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
W20260923 14:40:31.827529 25731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 14:40:31.827752 25731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 14:40:31.827803 25731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 14:40:31.827877 25731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 14:40:31.831274 25731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 14:40:31.831336 25731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 14:40:31.831375 25731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 14:40:31.831429 25731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 14:40:31.831460 25731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 14:40:31.835629 25731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.218.20:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.218.62:46435,127.2.218.61:34133,127.2.218.60:41957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/krb5kdc/kudu_127.2.218.60.keytab
--principal=kudu/127.2.218.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.218.60:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.218.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 23 Sep 2026 14:01:48 UTC on ccycloud.quasar-ievbqq.root.comops.site
build id 4667499
I20260923 14:40:31.836345 25731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 14:40:31.837114 25731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 14:40:31.844172 25737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:31.844168 25736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 14:40:31.844193 25739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 14:40:31.844476 25731 server_base.cc:1034] running on GCE node
I20260923 14:40:31.845021 25731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 14:40:31.845976 25731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 14:40:31.847160 25731 hybrid_clock.cc:648] HybridClock initialized: now 1790174431847154 us; error 28 us; skew 500 ppm
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 14:40:31.849514 25731 init.cc:377] Logged in from keytab as kudu/127.2.218.60@KRBTEST.COM (short username kudu)
I20260923 14:40:31.850339 25731 webserver.cc:460] Webserver started at http://127.2.218.60:45853/ using document root <none> and password file <none>
I20260923 14:40:31.850796 25731 fs_manager.cc:359] Metadata directory not provided
I20260923 14:40:31.850852 25731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 14:40:31.851061 25731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 14:40:31.851630 25724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } attempt: 2
I20260923 14:40:31.852166 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.851957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:59998)
0923 14:40:31.852065 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.852114 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 14:40:31.852785 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.852627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:60004)
0923 14:40:31.852708 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.852743 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 14:40:31.852960 25724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:41957: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
I20260923 14:40:31.853091 25731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/data/instance:
uuid: "00533a8c240c49a489ab006cbf9e4153"
format_stamp: "Formatted at 2026-09-23 14:40:31 on dist-test-slave-zl47"
server_key: "fd5515e2d33ed71f0dfbaf02384c8033"
server_key_iv: "16f90099c7cc1d690a0fb262b3bdc0dc"
server_key_version: "encryptionkey@0"
I20260923 14:40:31.853621 25731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/wal/instance:
uuid: "00533a8c240c49a489ab006cbf9e4153"
format_stamp: "Formatted at 2026-09-23 14:40:31 on dist-test-slave-zl47"
server_key: "fd5515e2d33ed71f0dfbaf02384c8033"
server_key_iv: "16f90099c7cc1d690a0fb262b3bdc0dc"
server_key_version: "encryptionkey@0"
I20260923 14:40:31.856662 25731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260923 14:40:31.858585 25746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 14:40:31.859665 25731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260923 14:40:31.859788 25731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/data,/tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/wal
uuid: "00533a8c240c49a489ab006cbf9e4153"
format_stamp: "Formatted at 2026-09-23 14:40:31 on dist-test-slave-zl47"
server_key: "fd5515e2d33ed71f0dfbaf02384c8033"
server_key_iv: "16f90099c7cc1d690a0fb262b3bdc0dc"
server_key_version: "encryptionkey@0"
I20260923 14:40:31.859880 25731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 14:40:31.869333 25731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 14:40:31.871641 25731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 14:40:31.871809 25731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 14:40:31.874864 25649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } attempt: 2
I20260923 14:40:31.875463 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.875225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:60010)
0923 14:40:31.875338 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.875388 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 14:40:31.876114 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.875932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:60018)
0923 14:40:31.876025 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.876064 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 14:40:31.876261 25649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.218.60:41957: Network error: Client connection negotiation failed: client connection to 127.2.218.60:41957: connect: Connection refused (error 111)
I20260923 14:40:31.877877 25731 rpc_server.cc:307] RPC server started. Bound to: 127.2.218.60:41957
I20260923 14:40:31.878157 25798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.218.60:41957 every 8 connection(s)
I20260923 14:40:31.878688 25731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/data/info.pb
I20260923 14:40:31.878993 2920 external_mini_cluster.cc:1388] Started /tmp/dist-test-task3TRGqf/build/debug/bin/kudu as pid 25731
I20260923 14:40:31.879119 2920 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task3TRGqf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790174173335390-2920-0/minicluster-data/master-2/wal/instance
I20260923 14:40:31.879362 2920 external_mini_cluster.cc:1428] Setting key d77f3fc8f914fd3527d185281266aa19
I20260923 14:40:31.881165 25799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 14:40:31.884024 25799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
I20260923 14:40:31.891233 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.880455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60030 (local address 127.2.218.60:41957)
0923 14:40:31.880805 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.880817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.880855 (+ 38us) server_negotiation.cc:407] Connection header received
0923 14:40:31.881435 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.881449 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.881703 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.881981 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.882679 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.883510 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.884261 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.884698 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.884847 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.887344 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.887349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.887358 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.887389 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.888793 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.889179 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.889184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.889188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.889293 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.889572 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.889575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.889576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.889850 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.890034 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.890270 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.890488 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":118,"threads_started":1}
I20260923 14:40:31.893085 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.885270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.2.218.62:46435)
0923 14:40:31.885370 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.885374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.885678 (+ 304us) server_negotiation.cc:407] Connection header received
0923 14:40:31.885784 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.885789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.885843 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.885937 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.886646 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.887124 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.888269 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.888450 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.888549 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.890383 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.890386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.890388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.890410 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.891624 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.892007 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.892010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.892011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.892092 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.892341 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.892343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.892344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.892496 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.892562 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.892866 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.892963 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 14:40:31.893111 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.885270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.62:46435 (local address 127.0.0.1:59468)
0923 14:40:31.885560 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.885598 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:31.885699 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:31.885980 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:31.885983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:31.886362 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:31.886509 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.886519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.887247 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.887250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.887977 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.887986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.888554 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.888557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.888657 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.889086 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:31.889101 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:31.890266 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:31.891727 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.891733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.891737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.891918 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.892180 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.892182 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.892184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.892263 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.892584 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:31.892588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:31.892694 (+ 106us) client_negotiation.cc:769] Sending connection context
0923 14:40:31.892814 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 14:40:31.892957 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":111,"threads_started":1}
I20260923 14:40:31.894181 25799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.62@KRBTEST.COM
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
I20260923 14:40:31.901149 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.893176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.2.218.62:46435)
0923 14:40:31.893257 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.893261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.893516 (+ 255us) server_negotiation.cc:407] Connection header received
0923 14:40:31.893595 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.893617 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.893655 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.893723 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.894421 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.894985 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.895638 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.895901 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.895994 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.898043 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.898046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.898049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.898071 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.899622 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.900067 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.900069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.900070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.900137 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.900401 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.900403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.900404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.900554 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.900765 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.900887 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.901012 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 14:40:31.901858 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.894584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.61:34133 (local address 127.0.0.1:47526)
0923 14:40:31.895110 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.895120 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:31.895201 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:31.895412 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:31.895415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:31.895560 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:31.895669 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.895675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.896387 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.896391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.896865 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.896874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.897302 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.897305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.897396 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.897760 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:31.897771 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:31.898877 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:31.900479 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.900482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.900484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.900651 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.900911 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.900914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.900916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.901007 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.901434 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:31.901437 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:31.901519 (+ 82us) client_negotiation.cc:769] Sending connection context
0923 14:40:31.901617 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 14:40:31.901698 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476}
I20260923 14:40:31.901935 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.894644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.2.218.61:34133)
0923 14:40:31.894744 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.894748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.895174 (+ 426us) server_negotiation.cc:407] Connection header received
0923 14:40:31.895270 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.895273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.895310 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.895394 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.895782 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.896252 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.896989 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.897199 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.897291 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.898995 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.898998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.899000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.899022 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.900401 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.900750 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.900753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.900755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.900826 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.901148 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.901152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.901154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.901341 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.901424 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.901686 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.901791 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 14:40:31.902648 25799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.60@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.218.61@KRBTEST.COM
I20260923 14:40:31.909606 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.903059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:60040)
0923 14:40:31.903139 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.903151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:31.903213 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:31.903367 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:31.903368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:31.903517 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:31.903586 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.903592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.904237 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.904239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.904694 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.904701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.905158 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.905160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.905268 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.905646 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:31.905658 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:31.906800 (+ 1142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:31.908403 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.908405 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.908407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.908568 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.908796 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.908800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.908802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.908871 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.909161 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:31.909163 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:31.909227 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 14:40:31.909323 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 14:40:31.909442 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":7}
I20260923 14:40:31.910013 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.903107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60040 (local address 127.2.218.60:41957)
0923 14:40:31.903209 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.903215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.903228 (+ 13us) server_negotiation.cc:407] Connection header received
0923 14:40:31.903257 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.903259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.903290 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.903362 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.903685 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.904145 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.904788 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.905058 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.905157 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.906948 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.906952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.906954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.906975 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.908289 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.908653 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.908658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.908660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.908721 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.908952 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.908954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.908955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.909082 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.909168 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.909843 (+ 675us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.909937 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 14:40:31.910493 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.902487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.2.218.61:34133)
0923 14:40:31.902572 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.902575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.902590 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:40:31.902710 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.902714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.902755 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.902826 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.903464 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.903946 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.904629 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.904906 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.904973 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.907568 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.907571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.907574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.907595 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.908912 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.909276 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.909280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.909281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.909353 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.909621 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.909624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.909625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.909808 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.910018 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.910157 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.910255 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 14:40:31.912376 25799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153: Bootstrap starting.
I20260923 14:40:31.913237 2920 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 14:40:31.914531 25799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:31.914264 25623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "88f61bb033b94f289e4e9aa80564e0b1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59476
I20260923 14:40:31.915234 25799 log.cc:824] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:31.916712 25799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153: No bootstrap required, opened a new log
I20260923 14:40:31.919155 25799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } }
I20260923 14:40:31.919337 25799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00533a8c240c49a489ab006cbf9e4153, State: Initialized, Role: FOLLOWER
I20260923 14:40:31.919607 25799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } }
I20260923 14:40:31.919993 25805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } } }
I20260923 14:40:31.920106 25805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:31.920657 25799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 14:40:31.924937 25816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:40:31.925004 25816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:40:31.924090 25731 master_runner.cc:186] Error getting master registration for 127.2.218.62:46435: OK, instance_id { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" instance_seqno: 1790174431521463 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:40:31.927086 25731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:40:31.944087 25724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } attempt: 3
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.61@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
I20260923 14:40:31.952004 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.944479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:60046)
0923 14:40:31.944574 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.944585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:31.944650 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:31.944896 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:31.944899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:31.945046 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:31.945124 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.945129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.945954 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.945957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.946679 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.946689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.947209 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.947213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.947308 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.947731 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:31.947744 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:31.948988 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:31.950694 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.950697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.950699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.950910 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.951166 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.951169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.951171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.951247 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.951604 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:31.951608 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:31.951684 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 14:40:31.951773 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 14:40:31.951881 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 14:40:31.952067 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.944513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.2.218.60:41957)
0923 14:40:31.944603 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.944607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.944622 (+ 15us) server_negotiation.cc:407] Connection header received
0923 14:40:31.944750 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.944753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.944789 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.944870 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.945233 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.945852 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.946802 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.947076 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.947152 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.949127 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.949130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.949134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.949162 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.950594 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.950999 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.951002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.951003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.951078 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.951330 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.951332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.951334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.951516 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.951601 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.951832 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.951925 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 14:40:31.954633 25724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9: Bootstrap starting.
I20260923 14:40:31.956334 25724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:31.956960 25724 log.cc:824] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:31.958554 25724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9: No bootstrap required, opened a new log
I20260923 14:40:31.960965 25724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } }
I20260923 14:40:31.961163 25724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f228f0a110548be9bcfa94207ce89c9, State: Initialized, Role: FOLLOWER
I20260923 14:40:31.961438 25724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } }
I20260923 14:40:31.961834 25819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } } }
I20260923 14:40:31.961967 25819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:31.962507 25724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 14:40:31.966781 25830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:40:31.966869 25830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:40:31.965611 25656 master_runner.cc:186] Error getting master registration for 127.2.218.62:46435: OK, instance_id { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" instance_seqno: 1790174431521463 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 14:40:31.968564 25656 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 14:40:31.975389 25649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } attempt: 3
Sep 23 14:40:31 dist-test-slave-zl47 krb5kdc[25568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790174431, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.218.62@KRBTEST.COM for kudu/127.2.218.60@KRBTEST.COM
I20260923 14:40:31.983289 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.975729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.218.60:41957 (local address 127.0.0.1:60062)
0923 14:40:31.975888 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 14:40:31.975901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 14:40:31.975982 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 14:40:31.976200 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 14:40:31.976202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 14:40:31.976370 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0923 14:40:31.976464 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.976471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.977288 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.977291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.977988 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 14:40:31.977998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.978486 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.978490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 14:40:31.978572 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.978992 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 14:40:31.979006 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 14:40:31.980230 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 14:40:31.981942 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.981945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.981946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.982190 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.982452 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 14:40:31.982455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 14:40:31.982456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 14:40:31.982529 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 14:40:31.982887 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 14:40:31.982889 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 14:40:31.982978 (+ 89us) client_negotiation.cc:769] Sending connection context
0923 14:40:31.983064 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 14:40:31.983164 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260923 14:40:31.983532 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.975787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.2.218.60:41957)
0923 14:40:31.975866 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.975869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.975961 (+ 92us) server_negotiation.cc:407] Connection header received
0923 14:40:31.976073 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.976075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:31.976109 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:31.976172 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:31.976573 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.977176 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.978123 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:31.978403 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:31.978479 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:31.980352 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:31.980356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:31.980358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:31.980381 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:31.981837 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.982276 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.982279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.982281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.982374 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:31.982616 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:31.982619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:31.982621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:31.982799 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:31.982899 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:31.983131 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 14:40:31.983224 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 14:40:31.985857 25649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1: Bootstrap starting.
I20260923 14:40:31.988096 25649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1: Neither blocks nor log segments found. Creating new log.
I20260923 14:40:31.988785 25649 log.cc:824] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1: Log is configured to *not* fsync() on all Append() calls
I20260923 14:40:31.990526 25649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1: No bootstrap required, opened a new log
I20260923 14:40:31.992969 25649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } }
I20260923 14:40:31.993196 25649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f61bb033b94f289e4e9aa80564e0b1, State: Initialized, Role: FOLLOWER
I20260923 14:40:31.993494 25649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } }
I20260923 14:40:31.994036 25833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } } }
I20260923 14:40:31.994243 25833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:31.994710 25649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 14:40:31.994796 25623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 14:40:31.994906 25623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:31.995944 25623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } }
I20260923 14:40:31.996723 25623 leader_election.cc:288] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [CANDIDATE]: Term 1 election: Requested vote from peers 4f228f0a110548be9bcfa94207ce89c9 (127.2.218.61:34133), 00533a8c240c49a489ab006cbf9e4153 (127.2.218.60:41957)
I20260923 14:40:31.997378 25698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88f61bb033b94f289e4e9aa80564e0b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4f228f0a110548be9bcfa94207ce89c9"
I20260923 14:40:31.997390 25773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88f61bb033b94f289e4e9aa80564e0b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "00533a8c240c49a489ab006cbf9e4153"
I20260923 14:40:31.997573 25698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:31.997573 25773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [term 0 FOLLOWER]: Advancing to term 1
I20260923 14:40:31.998803 25698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88f61bb033b94f289e4e9aa80564e0b1 in term 1.
I20260923 14:40:31.998803 25773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88f61bb033b94f289e4e9aa80564e0b1 in term 1.
I20260923 14:40:32.000099 25600 leader_election.cc:302] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00533a8c240c49a489ab006cbf9e4153, 88f61bb033b94f289e4e9aa80564e0b1; no voters:
I20260923 14:40:32.000561 25833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 14:40:32.000766 25833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [term 1 LEADER]: Becoming Leader. State: Replica: 88f61bb033b94f289e4e9aa80564e0b1, State: Running, Role: LEADER
I20260923 14:40:32.001062 25833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } }
I20260923 14:40:32.002756 25833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f61bb033b94f289e4e9aa80564e0b1. Latest consensus state: current_term: 1 leader_uuid: "88f61bb033b94f289e4e9aa80564e0b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } } }
I20260923 14:40:32.002898 25833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:32.004519 25581 master_runner.cc:263] No leader master found from master 88f61bb033b94f289e4e9aa80564e0b1
I20260923 14:40:32.004580 25581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 14:40:32.006398 25844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 14:40:32.006503 25844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 14:40:32.006599 25853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 14:40:32.007628 25853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 14:40:32.010221 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.998783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.2.218.60:41957)
0923 14:40:31.998891 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.998895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:31.999815 (+ 920us) server_negotiation.cc:407] Connection header received
0923 14:40:31.999993 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:31.999996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.000059 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.000113 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.001153 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:32.001811 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:32.003623 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:32.003913 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:32.003990 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:32.005699 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:32.005702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:32.005705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:32.005728 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:32.008136 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:32.008700 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:32.008703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:32.008705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:32.008785 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:32.009099 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:32.009102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:32.009104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:32.009296 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:32.009404 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:32.009954 (+ 550us) server_negotiation.cc:299] Negotiation successful
0923 14:40:32.010055 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260923 14:40:32.018608 25602 tablet.cc:2369] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 14:40:32.018993 25773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [term 1 FOLLOWER]: Refusing update from remote peer 88f61bb033b94f289e4e9aa80564e0b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:32.019580 25851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:32.020409 25698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [term 1 FOLLOWER]: Refusing update from remote peer 88f61bb033b94f289e4e9aa80564e0b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 14:40:32.021030 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:32.003146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.2.218.62:46435)
0923 14:40:32.003256 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.003262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.011343 (+ 8081us) server_negotiation.cc:407] Connection header received
0923 14:40:32.011493 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.011496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.011536 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.011582 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.012294 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:32.012937 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:32.013966 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:32.014266 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:32.014336 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:32.016274 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:32.016279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:32.016281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:32.016306 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:32.018484 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:32.018951 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:32.018955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:32.018957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:32.019040 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:32.019323 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:32.019326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:32.019328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:32.019514 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:32.019617 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:32.020772 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0923 14:40:32.020891 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 14:40:32.022727 25805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f61bb033b94f289e4e9aa80564e0b1. Latest consensus state: current_term: 1 leader_uuid: "88f61bb033b94f289e4e9aa80564e0b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } } }
I20260923 14:40:32.022830 25805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:32.022938 25851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 14:40:32.025852 25819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f61bb033b94f289e4e9aa80564e0b1. Latest consensus state: current_term: 1 leader_uuid: "88f61bb033b94f289e4e9aa80564e0b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } } }
I20260923 14:40:32.025976 25819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:32.030328 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:31.999498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.2.218.61:34133)
0923 14:40:31.999614 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:31.999619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.004685 (+ 5066us) server_negotiation.cc:407] Connection header received
0923 14:40:32.006171 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.006174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.006219 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.006280 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.016664 (+ 10384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:32.017303 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:32.022562 (+ 5259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:32.022869 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:32.022944 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:32.027025 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 14:40:32.027029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 14:40:32.027031 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 14:40:32.027056 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 14:40:32.028563 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:32.029040 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:32.029044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:32.029045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:32.029121 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 14:40:32.029427 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 14:40:32.029430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 14:40:32.029431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 14:40:32.029633 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 14:40:32.029749 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:32.030062 (+ 313us) server_negotiation.cc:299] Negotiation successful
0923 14:40:32.030190 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 14:40:32.033254 25851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88f61bb033b94f289e4e9aa80564e0b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } } }
I20260923 14:40:32.033349 25851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:32.034057 25833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88f61bb033b94f289e4e9aa80564e0b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } } }
I20260923 14:40:32.034173 25833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1 [sys.catalog]: This master's current role is: LEADER
I20260923 14:40:32.034310 25805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88f61bb033b94f289e4e9aa80564e0b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } } }
I20260923 14:40:32.034411 25805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:32.035039 25857 mvcc.cc:205] Tried to move back new op lower bound from 7332554473538760704 to 7332554473479368704. Current Snapshot: MvccSnapshot[applied={T|T < 7332554473538760704}]
I20260923 14:40:32.035084 25819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88f61bb033b94f289e4e9aa80564e0b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88f61bb033b94f289e4e9aa80564e0b1" member_type: VOTER last_known_addr { host: "127.2.218.62" port: 46435 } } peers { permanent_uuid: "4f228f0a110548be9bcfa94207ce89c9" member_type: VOTER last_known_addr { host: "127.2.218.61" port: 34133 } } peers { permanent_uuid: "00533a8c240c49a489ab006cbf9e4153" member_type: VOTER last_known_addr { host: "127.2.218.60" port: 41957 } } }
I20260923 14:40:32.035177 25819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 14:40:32.035632 25859 mvcc.cc:205] Tried to move back new op lower bound from 7332554473538760704 to 7332554473479368704. Current Snapshot: MvccSnapshot[applied={T|T < 7332554473538760704}]
I20260923 14:40:32.046438 25853 catalog_manager.cc:1347] Generated new cluster ID: 49c444b8bd0b44539525d83fb19e3b1c
I20260923 14:40:32.046490 25853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 14:40:32.067215 25853 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 14:40:32.068383 25853 catalog_manager.cc:1504] Loading token signing keys...
I20260923 14:40:32.086337 25853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88f61bb033b94f289e4e9aa80564e0b1: Generated new TSK 0
I20260923 14:40:32.087069 25853 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 14:40:32.118695 25653 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.117629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.2.218.62:46435)
0923 14:40:32.117718 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.117722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.117733 (+ 11us) server_negotiation.cc:407] Connection header received
0923 14:40:32.117771 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.117774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.117814 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.117881 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.118574 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59494: BlockingRecv error: recv got EOF from 127.0.0.1:59494 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260923 14:40:32.118918 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.117729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.2.218.61:34133)
0923 14:40:32.117821 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.117825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.117835 (+ 10us) server_negotiation.cc:407] Connection header received
0923 14:40:32.117871 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.117874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.117916 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.117975 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.118810 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47564: BlockingRecv error: recv got EOF from 127.0.0.1:47564 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260923 14:40:32.119700 25802 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.118255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.2.218.60:41957)
0923 14:40:32.118363 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.118367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.118965 (+ 598us) server_negotiation.cc:407] Connection header received
0923 14:40:32.119118 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.119121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.119163 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.119207 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.119621 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60078: BlockingRecv error: recv got EOF from 127.0.0.1:60078 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260923 14:40:32.123404 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.121422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.2.218.61:34133)
0923 14:40:32.121498 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.121503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.122332 (+ 829us) server_negotiation.cc:407] Connection header received
0923 14:40:32.122457 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.122459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.122517 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.122589 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.123343 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47572: BlockingRecv error: recv got EOF from 127.0.0.1:47572 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 14:40:32.126959 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.126287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.2.218.61:34133)
0923 14:40:32.126372 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.126376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.126388 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:40:32.126440 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.126442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.126479 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.126550 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.126885 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47578: BlockingRecv error: recv got EOF from 127.0.0.1:47578 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260923 14:40:32.131140 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.130432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.2.218.61:34133)
0923 14:40:32.130508 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.130512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.130549 (+ 37us) server_negotiation.cc:407] Connection header received
0923 14:40:32.130646 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.130648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.130680 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.130759 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.131071 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47584: BlockingRecv error: recv got EOF from 127.0.0.1:47584 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260923 14:40:32.137445 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.136553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.2.218.61:34133)
0923 14:40:32.136662 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.136665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.136840 (+ 175us) server_negotiation.cc:407] Connection header received
0923 14:40:32.136942 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.136945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.136995 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.137069 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.137378 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47592: BlockingRecv error: recv got EOF from 127.0.0.1:47592 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260923 14:40:32.147783 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.146915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.2.218.61:34133)
0923 14:40:32.146996 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.146999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.147179 (+ 180us) server_negotiation.cc:407] Connection header received
0923 14:40:32.147295 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.147298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.147338 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.147415 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.147717 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47600: BlockingRecv error: recv got EOF from 127.0.0.1:47600 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260923 14:40:32.165990 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.165113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.2.218.61:34133)
0923 14:40:32.165205 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.165209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.165395 (+ 186us) server_negotiation.cc:407] Connection header received
0923 14:40:32.165443 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.165446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.165489 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.165571 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.165904 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47614: BlockingRecv error: recv got EOF from 127.0.0.1:47614 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 14:40:32.200804 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.199848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.2.218.61:34133)
0923 14:40:32.199994 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.199996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.200073 (+ 77us) server_negotiation.cc:407] Connection header received
0923 14:40:32.200208 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.200210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.200248 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.200302 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.200705 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47616: BlockingRecv error: recv got EOF from 127.0.0.1:47616 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260923 14:40:32.267305 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.266310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.2.218.61:34133)
0923 14:40:32.266444 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.266448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.266608 (+ 160us) server_negotiation.cc:407] Connection header received
0923 14:40:32.266723 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.266726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.266768 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.266843 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.267203 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47622: BlockingRecv error: recv got EOF from 127.0.0.1:47622 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260923 14:40:32.397955 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.396989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.2.218.61:34133)
0923 14:40:32.397126 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.397130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.397248 (+ 118us) server_negotiation.cc:407] Connection header received
0923 14:40:32.397381 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.397384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.397430 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.397504 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.397856 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47638: BlockingRecv error: recv got EOF from 127.0.0.1:47638 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 14:40:32.656865 25729 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 14:40:32.655835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.2.218.61:34133)
0923 14:40:32.655977 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:32.655982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:32.656055 (+ 73us) server_negotiation.cc:407] Connection header received
0923 14:40:32.656210 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:32.656213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:32.656266 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:32.656345 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 14:40:32.656759 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47654: BlockingRecv error: recv got EOF from 127.0.0.1:47654 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 14:40:32.926074 25816 catalog_manager.cc:1259] Loaded cluster ID: 49c444b8bd0b44539525d83fb19e3b1c
I20260923 14:40:32.926158 25816 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153: loading cluster ID for follower catalog manager: success
I20260923 14:40:32.927584 25816 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153: acquiring CA information for follower catalog manager: success
I20260923 14:40:32.928610 25816 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00533a8c240c49a489ab006cbf9e4153: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:40:32.968005 25830 catalog_manager.cc:1259] Loaded cluster ID: 49c444b8bd0b44539525d83fb19e3b1c
I20260923 14:40:32.968065 25830 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9: loading cluster ID for follower catalog manager: success
I20260923 14:40:32.969592 25830 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9: acquiring CA information for follower catalog manager: success
I20260923 14:40:32.970706 25830 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f228f0a110548be9bcfa94207ce89c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 14:40:33.175025 25928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:33.171056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.2.218.61:34133)
0923 14:40:33.171312 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:33.171320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:33.171341 (+ 21us) server_negotiation.cc:407] Connection header received
0923 14:40:33.171531 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:33.171534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:33.171666 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:33.171795 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:33.172079 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:33.172570 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:33.173388 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:33.173655 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:33.173766 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:33.174180 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:33.174604 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:33.174745 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:33.174821 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 14:40:33.174872 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
I20260923 14:40:33.180653 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:33.177517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60082 (local address 127.2.218.60:41957)
0923 14:40:33.177697 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:33.177700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:33.177712 (+ 12us) server_negotiation.cc:407] Connection header received
0923 14:40:33.177782 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:33.177784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:33.177893 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:33.177978 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:33.178216 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:33.178728 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:33.179502 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:33.179684 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:33.179746 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:33.179966 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:33.180295 (+ 329us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:33.180379 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:33.180454 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 14:40:33.180494 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
I20260923 14:40:33.187295 25928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:33.183289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.2.218.61:34133)
0923 14:40:33.183390 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:33.183394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:33.183730 (+ 336us) server_negotiation.cc:407] Connection header received
0923 14:40:33.183920 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:33.183923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:33.183963 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:33.184004 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:33.184408 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:33.184947 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:33.186300 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:33.186587 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:33.186677 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:33.186931 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:33.186994 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:33.187058 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:33.187150 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 14:40:33.187193 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 14:40:33.187785 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:33.183385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.2.218.60:41957)
0923 14:40:33.183495 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:33.183499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:33.183834 (+ 335us) server_negotiation.cc:407] Connection header received
0923 14:40:33.183924 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:33.183927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:33.183964 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:33.184023 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:33.184338 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:33.185068 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:33.186738 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:33.186968 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:33.187065 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:33.187319 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:33.187420 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:33.187468 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:33.187630 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 14:40:33.187673 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 14:40:33.187850 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 14:40:33.183295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.2.218.62:46435)
0923 14:40:33.183531 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 14:40:33.183536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 14:40:33.183680 (+ 144us) server_negotiation.cc:407] Connection header received
0923 14:40:33.183910 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 14:40:33.183915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 14:40:33.184061 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 14:40:33.184159 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 14:40:33.184573 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:33.185303 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:33.186184 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 14:40:33.186515 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 14:40:33.186618 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 14:40:33.187163 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 14:40:33.187438 (+ 275us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 14:40:33.187544 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 14:40:33.187659 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 14:40:33.187710 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260923 14:40:33.190850 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 25581
I20260923 14:40:33.196244 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 25656
I20260923 14:40:33.201712 2920 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task3TRGqf/build/debug/bin/kudu with pid 25731
2026-09-23T14:40:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2830 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5716 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (259862 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST